Weird shape patterns produced when attempting to train on black & white medical data

[mckellwoodland]

Hello,

I am trying to train StyleGAN3 on 512x512 black and white PNG images. They are originally of abdominal CT scans with pixels blacked out far from the liver. Example images:

When I go to run StyleGAN3 on it, I receive images with weird shapes and lines inside the liver.

These images were taken at 21,600 kimg, but the same patterns can be seen throughout the entire training.

Command line:
./train.py --outdir /output --gpus 8 --data /input/rescaled255_outline.zip --cfg stylegan3-t --batch 32 --gamma 8.2 --betas 0.,0.99 --aug noaug --freezed 0 --p 0.2 --target 0.6 --cbase 32768 --cmax 512 --dlr 0.002 --mbstd-group 4 --metrics fid50k_full --kimg 25000 --tick 4 --snap 50 --seed 0 --fp32=1 --workers 3

The only thing I changed in the implementation was making the beta values a commandline parameter. Anyone have any ideas? I've tried setting beta0=0.9 with the same results. Also, tried changing gamma with the same results. When I run the styleGAN2 config, the lines/shapes don't show up. Though the noise is weird which is why I'm trying to get StyleGAN2 to work.

Thank you in advance!! :)

[Tollanador]

Is this potentially due to the aliasing artifacts resulting from improper anti-aliasing algorithms used in several common image processing libraries? When not processed correctly, a training image with lots of aliasing (potentially added during an image reprocessing step in your pipeline) can 'fool' the generator into kind of amplifying the repeating alias patterning and incorporating that into the various styles. That's if I understand the paper enough.

Essentially at the time of the paper being written, only PIL correctly performed several of the anti-aliasing algorithms.

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation

"When I run the styleGAN2 config, the lines/shapes don't show up."
The problem of poorly aliased training data only became apparent with StyleGan3 (vs StyleGan2) due to the changes in the architecture that deal with .. aliasing. :)

[PDillis]

As @Tollanador says, these are mostly artifacts of StyleGAN3, as you have two improvements regarding image manipulation you could say (translation and rotation invariance), but no improvement in image quality. If translation or rotation invariance doesn't matter to you (as with my limited medical knowledge, all CT scans will be aligned as the subjects are lying flat in the same position), then stick with StyleGAN2/ADA and e.g. transfer learn from BreCaHad or FFHQ512.

[mckellwoodland]

@Tollanador @PDillis

Thank you for the responses! All I do in my preprocessing pipeline is rescale voxel values to [0, 255] using Python mathematical operations and save 3D slices as 2D images using imageio. Do you think either of these could be the culprit? Or could it be that these anti-aliasing patterns are inherent to medical images?

And that is a good point about needing translational/rotational invariance in medical image. I've been working with StyleGAN2, but I haven't been able to get the liver texture right. When I look at images generated from the same latent vector across different training iterations, it has the same noise in the same spot, no matter if the anatomy looks completely different. It seemed similar to me to the texture sticking issues presented in the StyleGAN3 webpage, so I was hoping StyleGAN3 would fix the texture issues. In terms of, making the texture actually go with the rest of the liver instead of a fixed texture throughout training. Am I wrong in this assumption? Will translational invariance not help me? I'll include some StyleGAN2 images below to show you the texture problem (the fact that the texture isn't uniform in the liver).

[Tollanador]

I suppose, before you spend a lot of time wrestling with StyleGan, it may be useful to know what your intention is in using a GAN with medical imaging, as well as consider the features o the source data.

GANs effectively hallucinate an image, techniques exist to incorporate the source image as kind of structural anchor, this means that any output of a GAN contains hallucinated data that is relational to the training data. This is perfectly fine for creative applications such as generating faces, or style transfers, anything where accuracy isn't necessarily the highest of priority.
Medical Imaging though, generally needs very reliable, re-creatable, and highly accurate outputs as most Medical Imaging is used for diagnostic purposes.
It makes sense then, that you may want o avoid potentially hallucinating IN or OUT a disease.
For super-sampling purposes, GANs appear to work well with Medical Imaging, as the hallucinated data is carefully structured to give faithful upscaling of low resolution scanning images, however the more 'filled' in data (scaling multiplier) the higher the risk of introducing or masking disease clues.

As for the source data, think of it this way.

You are getting a 3D object, represented by a stack of 2D slices of pixel data.
StyleGan has no real sense of depth, there appears to be a gist of depth, however the StyleGan is enterally based on generating a single 2D image of similar qualities of the training data.
This means that any information contained in the 3D spatial arrangement of the original pixel data slices, is lost. The GAN might figure out some kind of 3D spatial arrangement-like .. representation, GANs have surprised before. This is more likely to happen, I would think, if the training data is of the same or limited, organs. So only datasets of CT abdominal scans, preferably from the same imaging device, with similar scanning parameters. If you have the DICOM datasets, you can also incorporate some of the parameters contained in the DICOM, such as the hardware-specific scanning parameters, try to avoid any parameters that are user-generated as they may not be accurate (due to the very large number of optional parameters in DICOM sets). This extra data could potentially be useful for the model as a kind of 'gating' that may prove to generate more accurate outputs taking into account the hardware source of the input image. For example, the noise patterns in the input images may be relational to the hardware used to acquire the image and the model may figure out multiple denoising-like algorithms based on the parameter inputs as well as the input image itself.

This information could be used by a properly designed architecture to more accurately processed a 2D slice for your purposes (or process it as a 3D object).

I presume this sort of stuff has been addressed by researchers creating models purely for medical imaging processing (even DICOM based), and multiple different techniques would be available. What, in your mind, does StyleGAN2/3 offer that potentially does better than other existing, massively trained models?

So it may be worthwhile to hunt around for a deep learning system that is designed for 3D spatial data as input, perhaps a point-cloud (or similar) based approach.

Apologies if this has all been considered already.

[agenoves]

Same problem in our hands than #81 and #77. Exact same type of patterns, and not due to image preprocessing as in our case we have none. Any objective consideration related to this problem would be welcome.

[mckellwoodland]

@Tollanador Thanks for the feedback. The idea is to use the GAN to model a training distribution for anomaly detection (see a review here). In the medical world, it is mostly use on low-dimensional histology/ophthalmology data. I'm interested in doing it in 512x512 to detect anatomical variations of test data from training data. But other GAN-based AD models perform poorly on that high of resolution, so I'm trying to use the StyleGAN architecture.

Anyway, I'm giving up on StyleGAN3. I did an extensive hyperparameter search and it always produced the same artifacts. Plus, I'm not convinced that rotation/translation invariance will fix my 2D "texture sticking issue". I'm just going to manually filter out the noise in the generated livers by StyleGAN2.

[matthewchung74]

Hi @mckellwoodland , I know you're not using stylegen but I was reading this post and wondering, with a single CT, there can be a lot of images. Were you planning on putting all the images of the ct on one image grid?

[mckellwoodland]

@matthewchung74 I'm only generating random 2D slices of 3D CT scans. When I generate 2D slices, the slices aren't necessarily supposed to be from the same scan or even patient. I prefer looking at single slices at a time so that I can see more fine-grained details. So no, I didn't present them in one grid for my work.

[matthewchung74]

Thanks for letting me know