MusicGen: Add Stereo Model

[sanchit-gandhi]

What does this PR do?

The original MusicGen model generates mono (1-channel) outputs. It does this by predicting a set of 4 codebooks at each generation step:

[codebook_1, codebook_2, codebook_3, codebook_4]

After generating, the sequence of predicted codebooks is passed through the EnCodec model to get the final waveform.

This PR adds the MusicGen stereo model. It works by predicting two sets of codebooks at each step. One set of codebooks corresponds to the left channel, the other set corresponds to the right channel. The sets of codebooks are interleaved as follows:

[left_codebook_1, right_codebook_1, left_codebook_2, right_codebook_2, ..., left_codebook_4, right_codebook_4]

After generating, the sequence of generated codebooks are partitioned into their left/right parts, and then each sequence passed through EnCodec to get the left/right waveform respectively.

[sanchit-gandhi]

Following the EnCodec naming here:

transformers/src/transformers/models/encodec/configuration_encodec.py

Lines 50 to 51 in d7cb5e1

	audio_channels (`int`, *optional*, defaults to 1):
	Number of channels in the audio data. Either 1 for mono or 2 for stereo.

Note that the EnCodec model used is still 1-channel (mono) - it's just the MusicGen model that works in a 2-channel fashion.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ArthurZucker]

Thanks looks very clean 😉

[ArthurZucker]

Is the format we gave (interleaving) forced by BC? Otherwise storing in two list or tuples would be better IMO (kind of like how audio are stored no?)
my only "complain" here

[sanchit-gandhi]

The original model is designed to predict them in an interleaved way:

[left_codebook_1, right_codebook_1, left_codebook_2, right_codebook_2, ..., left_codebook_4, right_codebook_4]

We could change this to predict left first, then right:

[left_codebook_1, left_codebook_2, ..., left_codebook_4, right_codebook_1, right_codebook_2, ..., right_codebook_4]

Which would require re-shaping the LM head weights, and duplicating the pattern mask along the row dimension. Overall I think the complexity would be similar to the interleaved way we have now.

But predicting two sets of codebooks as two separate tuples would break compatibility with the existing mono musicgen, or otherwise complicate the code since we'll have different inputs / sampling logic depending on whether we're mono / stereo.

[ArthurZucker]

Awesome thanks for explaining