Minimal Implimentation of VCRec (2024) for collapse prevention.
/layer.py Direct Gradient Manipulation implimentation of VCReg. Put this between your blocks. Assumes 0 mean.
/loss.py VICReg loss, for applying this regularization to an output representation.
Original Paper:
@misc{zhu2024variancecovariance,
title={Variance-Covariance Regularization Improves Representation Learning},
author={Jiachen Zhu and Katrina Evtimova and Yubei Chen and Ravid Shwartz-Ziv and Yann LeCun},
year={2024},
eprint={2306.13292},
archivePrefix={arXiv},
primaryClass={cs.LG}
}