'Hubert' PyTorch CUDA Memory Error

[rjchristopher27]

Running 'Hubert' cell results in CUDA memory error. Adjusting batch size in config.json doesn't affect resulting memory allocation. Smaller dataset doesn't change anything either. Issue seems to lie in PyTorch with its excessive use of VRAM. Any guidance would be greatly appreciated.

[96]
Loading hubert for content...
load model(s) from hubert/checkpoint_best_legacy_500.pt
INFO:fairseq.tasks.hubert_pretraining:current directory is /content/so-vits-svc
INFO:fairseq.tasks.hubert_pretraining:HubertPretrainingTask Config {'_name': 'hubert_pretraining', 'data': 'metadata', 'fine_tuning': False, 'labels': ['km'], 'label_dir': 'label', 'label_rate': 50.0, 'sample_rate': 16000, 'normalize': False, 'enable_padding': False, 'max_keep_size': None, 'max_sample_size': 250000, 'min_sample_size': 32000, 'single_target': False, 'random_crop': True, 'pad_audio': False}
INFO:fairseq.models.hubert.hubert:HubertModel Config: {'_name': 'hubert', 'label_rate': 50.0, 'extractor_mode': default, 'encoder_layers': 12, 'encoder_embed_dim': 768, 'encoder_ffn_embed_dim': 3072, 'encoder_attention_heads': 12, 'activation_fn': gelu, 'layer_type': transformer, 'dropout': 0.1, 'attention_dropout': 0.1, 'activation_dropout': 0.0, 'encoder_layerdrop': 0.05, 'dropout_input': 0.1, 'dropout_features': 0.1, 'final_dim': 256, 'untie_final_proj': True, 'layer_norm_first': False, 'conv_feature_layers': '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2', 'conv_bias': False, 'logit_temp': 0.1, 'target_glu': False, 'feature_grad_mult': 0.1, 'mask_length': 10, 'mask_prob': 0.8, 'mask_selection': static, 'mask_other': 0.0, 'no_mask_overlap': False, 'mask_min_space': 1, 'mask_channel_length': 10, 'mask_channel_prob': 0.0, 'mask_channel_selection': static, 'mask_channel_other': 0.0, 'no_mask_channel_overlap': False, 'mask_channel_min_space': 1, 'conv_pos': 128, 'conv_pos_groups': 16, 'latent_temp': [2.0, 0.5, 0.999995], 'skip_masked': False, 'skip_nomask': False, 'checkpoint_activations': False, 'required_seq_len_multiple': 2, 'depthwise_conv_kernel_size': 31, 'attn_type': '', 'pos_enc_type': 'abs', 'fp16': False}
Loaded hubert.
73% 70/96 [00:50<00:18, 1.39it/s]
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/content/so-vits-svc/preprocess_hubert_f0.py", line 44, in process_batch
process_one(filename, hmodel)
File "/content/so-vits-svc/preprocess_hubert_f0.py", line 30, in process_one
c = utils.get_hubert_content(hmodel, wav_16k_tensor=wav16k)
File "/content/so-vits-svc/utils.py", line 207, in get_hubert_content
logits = hmodel.extract_features(**inputs)
File "/usr/local/lib/python3.9/dist-packages/fairseq/models/hubert/hubert.py", line 535, in extract_features
res = self.forward(
File "/usr/local/lib/python3.9/dist-packages/fairseq/models/hubert/hubert.py", line 467, in forward
x, _ = self.encoder(
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/fairseq/models/wav2vec/wav2vec2.py", line 1003, in forward
x, layer_results = self.extract_features(x, padding_mask, layer)
File "/usr/local/lib/python3.9/dist-packages/fairseq/models/wav2vec/wav2vec2.py", line 1049, in extract_features
x, (z, lr) = layer(
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/fairseq/models/wav2vec/wav2vec2.py", line 1260, in forward
x, attn = self.self_attn(
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/fairseq/modules/multihead_attention.py", line 538, in forward
return F.multi_head_attention_forward(
File "/usr/local/lib/python3.9/dist-packages/torch/nn/functional.py", line 5161, in multi_head_attention_forward
attn_output_weights = softmax(attn_output_weights, dim=-1)
File "/usr/local/lib/python3.9/dist-packages/torch/nn/functional.py", line 1841, in softmax
ret = input.softmax(dim)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.86 GiB (GPU 0; 14.75 GiB total capacity; 8.55 GiB already allocated; 5.03 GiB free; 8.68 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[jagilley]

@rjchristopher27 this happens if your training files are too long. Use a utility like ffmpeg to trim them before running hubert