Variance-Covariance Regularization Improves Representation Learning

Minimal Implimentation of VCRec (2024) for collapse prevention.

/layer.py Direct Gradient Manipulation implimentation of VCReg. Put this between your blocks. Assumes 0 mean.

/loss.py VICReg loss, for applying this regularization to an output representation.

Original Paper:

@misc{zhu2024variancecovariance,
      title={Variance-Covariance Regularization Improves Representation Learning}, 
      author={Jiachen Zhu and Katrina Evtimova and Yubei Chen and Ravid Shwartz-Ziv and Yann LeCun},
      year={2024},
      eprint={2306.13292},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}