Add TensorFlow Wav2Vec2 for sequence classification #22073

nandwalritik · 2023-03-10T06:52:45Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case. Add TensorFlow Wav2Vec2 for sequence classification #21778
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2023-03-10T07:11:22Z

The documentation is not available anymore as the PR was closed or merged.

github-actions · 2023-04-09T15:02:27Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

sgugger · 2023-04-10T13:05:46Z

Kindly ping @sanchit-gandhi and adding @Rocketknight1 for the TensorFlow side.

Rocketknight1 · 2023-04-11T14:24:29Z

Hi @nandwalritik, and sorry for the extremely long delay in catching this! Ordinarily one of the TF maintainers reviews TF pull requests, but this one slipped through the cracks somehow. If you want to file TF PRs in future, you can directly ping me or @gante to make sure that we don't miss it.

This PR actually looks almost perfect, but there are a couple of TF-specific details that are causing some tests to fail. I'll mark them in a code review in just a sec, but they shouldn't take too long to fix. Thanks again for submitting this!

Rocketknight1

This looks good! A few tweaks in the __init__ should fix most of the issues.

The only other thing missing is a serving and serving_output method. These are properties that are unique to TF models, and indicate the input and output signatures to enable model compilation and exporting. You can look at other models in the library to get a sense for how they work, but if you can't figure it out let me know and I'll add it for you!

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py

nandwalritik · 2023-04-13T06:47:35Z

for serving and serving_output methods I added changes, but now sure they are correct or not.

Rocketknight1 · 2023-04-14T16:45:16Z

Hi @nandwalritik, I'm seeing the issue when you move it to build() - the problem is the weight name, as it usually is in our TensorFlow ports! TF isn't very consistent about the name scope used for weights, and it can differ depending on when the weight is created in the init, the build or lazily in the call(), which makes it tricky because we use the names to match weights between PT and TF models.

I'll see if I can push a solution to your repo, hang on.

nandwalritik · 2023-04-14T16:46:55Z

Ok

Rocketknight1 · 2023-04-14T18:31:09Z

Try:

with tf.name_scope(self._name_scope()):
    self.layer_weights = self.add_weight(
        shape=(self.num_layers,), initializer="ones", trainable=True, name="layer_weights"
    )

in the __init__, not the build(). I know that contradicts what I said earlier, but it turns out to be a bit different for a base model class than a sublayer.

I also see a couple of other errors - you can see them by clicking the Details beside tests_tf in the checklist at the bottom of this PR. If you can't figure out what's causing them, ping me over the weekend or on Monday and I'll try to debug them!

nandwalritik · 2023-04-17T04:58:14Z

Try:
with tf.name_scope(self._name_scope()):
    self.layer_weights = self.add_weight(
        shape=(self.num_layers,), initializer="ones", trainable=True, name="layer_weights"
    )
in the __init__, not the build(). I know that contradicts what I said earlier, but it turns out to be a bit different for a base model class than a sublayer.

I also see a couple of other errors - you can see them by clicking the Details beside tests_tf in the checklist at the bottom of this PR. If you can't figure out what's causing them, ping me over the weekend or on Monday and I'll try to debug them!

Ok, so after adding this change, the weights are getting loaded without any warning or error, but the output of pytorch and tensorflow model doesn't have rtol of 1e-5.
Although I checked shape and absolute sum of tensors of both the models they are almost equal

PT model 
1,292,768 -> 29877.8750


1,292,256 -> 29711.7109

pooled_output
1,256 -> 38.7491



TF model

hidden_state
1,292,768 -> 29877.879

1,292,256 -> 29711.715

pooled_output
1,256 -> 38.811996

What should i try next to satisfy rtol criteria.

Rocketknight1 · 2023-04-17T16:49:26Z

Hm, those are some fairly large discrepancies! The debugging process we recommend when something like that happens is:

Make a test environment and load the PT and TF models with the same weights
Try to isolate the earliest point where the model outputs diverge. You can use options like output_hidden_states to get the model to return all hidden states, not just the final ones.
Once you find the first point of divergence, try to see if you can dig into the layer where the divergence happened. You can place breakpoints, or extract sublayers and try passing test inputs into them.
Eventually you will find the single specific place where the divergence creeps in - now you can check what the cause is. Make sure the weights for that operation really do match between the two frameworks, and make sure both frameworks are doing the same thing at that point.

As always, if you can't figure it out, let me know! This kind of work can be quite gruelling, but we really appreciate the work you're doing on the model port.

nandwalritik · 2023-04-21T06:36:54Z

Hi @Rocketknight1 I added test cases and fixed the feed forward part, but the CI is failing due to flax, I think this might not be related to my changes. Please review the PR and let me know if any more changes are required.

Rocketknight1 · 2023-04-21T13:44:03Z

Yep, those flax issues are unrelated, just ignore them. I'll review everything today, but the CI looks good!

Rocketknight1

This looks really good! I left a couple of minor comments, but this is basically ready to go at this point. The inference tests give me very high confidence that this matches the behaviour of the PT/FLAX model up to numerical error. Thanks for all the effort you put in with this PR - it's really appreciated, and I think people will get a lot of use out of the result!

tests/models/wav2vec2/test_modeling_tf_wav2vec2.py

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py

sanchit-gandhi

Very nice PR @nandwalritik - looks good from an audio perspective. Just wanted to confirm that the slow tests pass? I'm pretty confident we have equality between the TF and PT code based on what I've seen, but would love to hear what @Rocketknight1 says here too!

Edit: he beat me to it!

sanchit-gandhi · 2023-04-21T13:49:58Z

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py

+            (batch_size, feature_vector_length), dtype=attention_mask.dtype, name="attention_mask"
+        )  # these two operations makes sure that all values before the output lengths idxs are attended to
+        ## check device
+        attention_mask = tf.tensor_scatter_nd_update(


Looks good to me - will leave the TF specificities to @Rocketknight1!

This is correct, as far as I can see! tensor_scatter_nd_update is TF's equivalent to JAX's array assignment with the .at[].set() operation.

sanchit-gandhi · 2023-04-21T13:57:19Z

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py

@@ -1552,3 +1592,123 @@ def serving_output(self, output: TFCausalLMOutput) -> TFCausalLMOutput:
        hidden_states = tf.convert_to_tensor(output.hidden_states) if self.config.output_hidden_states else None
        attentions = tf.convert_to_tensor(output.attentions) if self.config.output_attentions else None
        return TFCausalLMOutput(logits=output.logits, hidden_states=hidden_states, attentions=attentions)
+
+
+class TFWav2Vec2ForSequenceClassification(TFWav2Vec2PreTrainedModel):


Also looks good to me (matching PyTorch) - will leave the TF specificities again to @Rocketknight1

sanchit-gandhi · 2023-04-21T13:59:07Z

tests/models/wav2vec2/test_modeling_tf_wav2vec2.py

+    def test_inference_keyword_spotting(self):
+        model = TFWav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-base-superb-ks", from_pt=True)
+        processor = AutoFeatureExtractor.from_pretrained("superb/wav2vec2-base-superb-ks")
+        input_data = self._load_superb("ks", 4)


Super(b)! Excellent work on getting the slow tests to pass!

nandwalritik · 2023-04-26T04:51:09Z

@sanchit-gandhi @Rocketknight1 let me know if any more changes are required or else can you guys get this pr merged.

Rocketknight1 · 2023-04-26T12:35:21Z

Just looked over the last few changes - I'm happy to merge it at this point. Thanks again for putting in the work on this!

* Add initial changes for TF wav2vec2 for sequence classification * Add suggested changes * Add serving and serving output methods * Add serving_output implementation and fix layer_weights * Add fixes * Fixed test cases * Fixing test and adding suggested changes

nandwalritik mentioned this pull request Mar 10, 2023

Add TensorFlow Wav2Vec2 for sequence classification #21778

Open

Add initial changes for TF wav2vec2 for sequence classification

d3f74c2

Rocketknight1 reviewed Apr 11, 2023

View reviewed changes

Add suggested changes

8252b9f

nandwalritik force-pushed the add_wav2vec2_seq_classification branch from 94d74f8 to 8252b9f Compare April 12, 2023 04:46

Add serving and serving output methods

592d47f

Add serving_output implementation and fix layer_weights

983965e

nandwalritik force-pushed the add_wav2vec2_seq_classification branch from 3da119b to 983965e Compare April 17, 2023 04:51

nandwalritik added 2 commits April 19, 2023 16:45

Add fixes

c22aef5

Fixed test cases

7a92f7d

Rocketknight1 approved these changes Apr 21, 2023

View reviewed changes

tests/models/wav2vec2/test_modeling_tf_wav2vec2.py Outdated Show resolved Hide resolved

src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py Show resolved Hide resolved

sanchit-gandhi approved these changes Apr 21, 2023

View reviewed changes

Fixing test and adding suggested changes

e6080ac

Rocketknight1 merged commit 20ac86c into huggingface:main Apr 26, 2023

sanchit-gandhi mentioned this pull request Apr 26, 2023

Add tensor flow whisper model for audio classification #22109

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TensorFlow Wav2Vec2 for sequence classification #22073

Add TensorFlow Wav2Vec2 for sequence classification #22073

nandwalritik commented Mar 10, 2023

HuggingFaceDocBuilderDev commented Mar 10, 2023 •

edited

Loading

github-actions bot commented Apr 9, 2023

sgugger commented Apr 10, 2023

Rocketknight1 commented Apr 11, 2023

Rocketknight1 left a comment

nandwalritik commented Apr 13, 2023

Rocketknight1 commented Apr 14, 2023

nandwalritik commented Apr 14, 2023

Rocketknight1 commented Apr 14, 2023

nandwalritik commented Apr 17, 2023

Rocketknight1 commented Apr 17, 2023 •

edited

Loading

nandwalritik commented Apr 21, 2023

Rocketknight1 commented Apr 21, 2023

Rocketknight1 left a comment •

edited

Loading

sanchit-gandhi left a comment •

edited

Loading

sanchit-gandhi Apr 21, 2023

Rocketknight1 Apr 21, 2023

sanchit-gandhi Apr 21, 2023

sanchit-gandhi Apr 21, 2023

nandwalritik commented Apr 26, 2023

Rocketknight1 commented Apr 26, 2023

Add TensorFlow Wav2Vec2 for sequence classification #22073

Add TensorFlow Wav2Vec2 for sequence classification #22073

Conversation

nandwalritik commented Mar 10, 2023

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Mar 10, 2023 • edited Loading

github-actions bot commented Apr 9, 2023

sgugger commented Apr 10, 2023

Rocketknight1 commented Apr 11, 2023

Rocketknight1 left a comment

Choose a reason for hiding this comment

nandwalritik commented Apr 13, 2023

Rocketknight1 commented Apr 14, 2023

nandwalritik commented Apr 14, 2023

Rocketknight1 commented Apr 14, 2023

nandwalritik commented Apr 17, 2023

Rocketknight1 commented Apr 17, 2023 • edited Loading

nandwalritik commented Apr 21, 2023

Rocketknight1 commented Apr 21, 2023

Rocketknight1 left a comment • edited Loading

Choose a reason for hiding this comment

sanchit-gandhi left a comment • edited Loading

Choose a reason for hiding this comment

sanchit-gandhi Apr 21, 2023

Choose a reason for hiding this comment

Rocketknight1 Apr 21, 2023

Choose a reason for hiding this comment

sanchit-gandhi Apr 21, 2023

Choose a reason for hiding this comment

sanchit-gandhi Apr 21, 2023

Choose a reason for hiding this comment

nandwalritik commented Apr 26, 2023

Rocketknight1 commented Apr 26, 2023

HuggingFaceDocBuilderDev commented Mar 10, 2023 •

edited

Loading

Rocketknight1 commented Apr 17, 2023 •

edited

Loading

Rocketknight1 left a comment •

edited

Loading

sanchit-gandhi left a comment •

edited

Loading