Range for heuristic r_t and ADA issues

[hojaeklee]

According to page 5 of the ADA paper ("Training Generative Adversarial Networks with Limited Data"), it says "For both heuristics, r = 0 means no overfitting and r = 1 indicates complete overfitting,...".

I am using the r_t heuristic, but when I plot the values, it seems to range from [-1, 1]. From my understanding of the code, it seems like D_train is the output of a Linear layer, which can range (-inf, inf). So, it seems to make sense that the expectation of the sign(D_train) is [-1, 1].

I'm suspecting my ADA isn't working (i.e. not improving performance in terms of FID) because the heuristic seems to always be close to r_t = 1, and then sharply decline to r_t = -1.

I would love to get clarification on the range of r_t and whether someone has any suggestions to use ADA to improve GAN performance. I also noticed augmentation probability p always increases (above 1) as #27 pointed out, and I have tried both capping the "probability" at 0.8 and other suggestions from the issue.

Thank you very much!