Add Fourier Domain Augmentations

[guarin]

Add augmentations from Disentangling the Effects of Data Augmentation and Format Transform in Self-Supervised Learning of Image Representations.

Todo

• Add RFFT2 transform #1657
• Add IRFFT2 transform #1656
• Add AmplitudeRescale transform #1658
• Add PhaseShift transform #1659
• Add RandomFrequenceMask transform #1660
• Add GaussianMixtureMask transform #1661

Please create one PR per transform.

The following hyperparameters should be implemented:

The augmentations are explained in more detail in Section 4 of the paper. All augmentations should take a tensor as input and return a tensor again. Tensors must have shape (C, H, W). See https://github.com/lightly-ai/lightly/blob/master/lightly/transforms/rotation.py for an example implementation of a transform.

For unit tests please just assert that the expected shape of the output is correct. We don't require more tests because testing transforms is tricky. Ideally you can also create some example images where the transform was applied (take an image, load it with PIL, convert it to a tensor with ToTensor transform from torchvision, apply the newly implemented transform, covert back to image using ToPILImage and upload the final image). Best vary the hyperparameters a bit to see how they affect the final images.

[KekmaTime]

Hi @guarin , I'd like to work on this issue. Could you please guide me on where to create the necessary files? Knowing this will help streamline my work.

[guarin]

Hi! Thanks for looking into this!

Transforms should be added to lightly/transforms. The gaussian blur transform is a good example: https://github.com/lightly-ai/lightly/blob/master/lightly/transforms/gaussian_blur.py

Please also add minimal tests to tests/transforms. It can be as simple as calling the transform with some random image/tensor and check that it runs through and the output has the correct shape.

For the fourier transforms I would assume that the inputs are already tensors (the user added a ToTensor transform before).

For general info on how to contribute please have a look at: https://github.com/lightly-ai/lightly/blob/master/CONTRIBUTING.md

[payo101]

Hey @guarin , Is this issue still being worked on by someone else, or could I also look into it?

[guarin]

Hi! None of the transforms have been implemented so far, so all of them are still up for grabs. @KekmaTime did you already have time to look into this?

[KekmaTime]

@guarin hey. yes you can assign this to someone else as i've been a bit busy irl 👍🏽

[belloibrahv]

Hi @guarin,

I'm interested in contributing to this issue for implementing Fourier Domain Augmentations. I have experience with PyTorch and image processing, and I'd like to help implement one or more of the required transforms.

Given that there are multiple transforms needed and some contributors may have already started working, I have a few questions:

1. Are any of the following transforms still available to work on?

   • AmplitudeRescale
   • PhaseShift
   • RandomFrequenceMask
   • GaussianMixtureMask

2. For whichever transform is available, could you provide some guidance on:

   • Expected input/output tensor shapes
   • Any specific parameters that should be configurable
   • Preferred approach for handling edge cases

3. Regarding testing:

   • Should we follow the same testing pattern as in the gaussian_blur.py example?
   • Are there any specific test cases you'd like to see beyond shape checking?

I've reviewed the CONTRIBUTING.md and am prepared to follow the project's guidelines. I'd be happy to start with whichever transform you think would be most helpful to tackle next.

Thank you for considering my contribution!

[guarin]

Hi, I created issues for the different transforms. Please indicate which transform you would like to work on and I can assign the issue to you :)

Will add more info about the issues shortly.

[guarin]

@belloibrahv I added some more information to the issue description, hopefully this clarifies your questions :) Regarding edge cases I wouldn't worry too much, as long as it works for images with height and width >= 32 pixels it should be good. If there is anything you deem important you can mention it in the PR or leave a comment in the code.

I also contacted the authors to get access to the source code, this should help with the implementation.