Pass device in Logits Processor's init #29804

zucchini-nlp · 2024-03-22T09:52:42Z

What does this PR do?

This PR adds the ability to pass in device when initializing LogitsProcessors and is one more step towards compile compatibility.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante

HuggingFaceDocBuilderDev · 2024-03-22T10:13:17Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

gante

Overall notes before going to details:

In the processors that take eos_token_id as input: see Generate: consistently handle special tokens as tensors #29788. In this PR, the special tokens are treated as tensors by default, solving most of the needed changes. I would rebase this PR on main after that PR is merged, as some of the changes here will become redundant :)
On the processors that don't need to use device, such as TemperatureLogitsWarper -- let's not add unused arguments. Clean interfaces are important 🧼 (unless there are significant benefits from standardizing them)
Let's not throw a warning when the device is not passed and tensors are initialized on CPU. A .to operation is not that expensive :)

zucchini-nlp · 2024-03-25T15:41:50Z

Cool, I did not notice that
2 and 3. Okay, thought we need it for consistency like we had with other new args in public classes. Will remove it and rebase later

zucchini-nlp · 2024-04-22T08:06:34Z

Not stale

…vice

zucchini-nlp · 2024-05-09T20:22:51Z

This PR now can be reviewed. Rebased main and updated the changes. All the tests from RUN_SLOW=1 pytest tests/generation are passing on my end

gante

LGTM, thank you for improving generate :D

src/transformers/models/whisper/generation_whisper.py

Co-authored-by: Joao Gante <[email protected]>

…vice

zucchini-nlp · 2024-05-22T11:59:57Z

@gante Ah I forgot whisper is encoder-decoder. Oke, now it infers device from one of the inputs passed by the user.

ArthurZucker · 2024-05-23T17:02:58Z

How could the bot come 🤣 anyways on it!

ArthurZucker

Overall LGTM, not sure input_ids device is the always the best, and we need a small test to see which feature is enable by this potentially!

ArthurZucker · 2024-05-23T17:03:55Z

src/transformers/generation/logits_process.py

+        if device is None:
+            device = "cpu"
+


I'd argue that we can just set it to "cpu" in the arg no?

This is mostly for users who use/pass LogitsProcessor as a standalone kwarg, because 'generate()' takes care that device is not None.

I think we should raise warning for BC saying users to pass-in the device, but let's ask @gante if he's okay with it. If I am not misunderstanding, we shouldn't raise warnings 🤔

Let's not throw a warning when the device is not passed and tensors are initialized on CPU. A .to operation is not that expensive :)

yeah, don't think it's a problem to silently do this

Down to just default to CPU which was already the behaviour by default before this PR no?

ahh my bad, didn't read carefully the first comment. Setting in the arg as default is better, right

My concern is that before this PR, we were placing these on scores.device during "_ [call]_ " , but anyway I still get lost at when to do BC deprecation and when to not do 😄

transformers/src/transformers/generation/logits_process.py

Line 155 in 03935d3

self.eos_token_id = self.eos_token_id.to(scores.device)

ArthurZucker · 2024-05-23T17:04:39Z

src/transformers/generation/utils.py

@@ -1700,7 +1737,7 @@ def generate(
            encoder_input_ids=inputs_tensor,
            prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
            logits_processor=logits_processor,
-            device=inputs_tensor.device,
+            device=input_ids.device,


why is this required ?

Right! I thought that it was me who changed to inputs_tensor and was trying to revert 😆 I'll revert it back, no difference whichever tensor we use here

ArthurZucker · 2024-05-23T17:05:43Z

src/transformers/generation/utils.py

should be use self.device? or lm_head.device? (which is not always there but still)

ArthurZucker · 2024-05-23T17:06:12Z

tests/generation/test_logits_process.py

I think we need to make sure dive placement on multi GPU works, might already be tested !

Yes, I'll try to add a test if I can. But we can be quite sure device placement of inputs is the correct one, as discusses with @gante this PR recommends to use input's device and not model params in multiGPU setting

got it. Any how LGTM

ArthurZucker

could you rebase your branch ? (format changes seems unrelated?)

zucchini-nlp · 2024-06-04T05:19:11Z

Oke, rebased main and the unnecessary formatting is removed. Will merge as I guess we don't need to add warnings :)

* add device in logits processor * remove device when not needed * codestyle * tests * forgot `melody` version * Update src/transformers/models/whisper/generation_whisper.py Co-authored-by: Joao Gante <[email protected]> * codestyle * updates --------- Co-authored-by: Joao Gante <[email protected]>

add device in logits processor

5f0c073

gante reviewed Mar 25, 2024

View reviewed changes

zucchini-nlp added 3 commits May 9, 2024 22:00

Merge remote-tracking branch 'upstream/main' into logits_processor_de…

045e910

…vice

remove device when not needed

9f914c1

codestyle

052da72

zucchini-nlp requested a review from gante May 9, 2024 20:22

zucchini-nlp added 2 commits May 10, 2024 09:20

tests

cfaae6d

forgot melody version

7290bc1

gante approved these changes May 21, 2024

View reviewed changes

src/transformers/models/whisper/generation_whisper.py Outdated Show resolved Hide resolved

zucchini-nlp and others added 4 commits May 22, 2024 13:31

Update src/transformers/models/whisper/generation_whisper.py

5450a0a

Co-authored-by: Joao Gante <[email protected]>

Merge branch 'main' into logits_processor_device

22560b9

Merge remote-tracking branch 'upstream/main' into logits_processor_de…

7ef58b6

…vice

codestyle

7e1530a

zucchini-nlp requested a review from ArthurZucker May 22, 2024 12:00

huggingface deleted a comment from github-actions bot May 22, 2024

zucchini-nlp added 2 commits May 23, 2024 13:31

Merge branch 'huggingface:main' into logits_processor_device

25d71ee

Merge branch 'huggingface:main' into logits_processor_device

9c10511

ArthurZucker reviewed May 23, 2024

View reviewed changes

updates

ed3e3fa

ArthurZucker approved these changes Jun 3, 2024

View reviewed changes

Merge branch 'huggingface:main' into logits_processor_device

6fd8c9e

zucchini-nlp merged commit 83238ee into huggingface:main Jun 4, 2024
23 checks passed

ylacombe mentioned this pull request Jun 14, 2024

Fix Bark logits processors device misplacement #31416

Merged

ywang96 mentioned this pull request Jun 27, 2024

[CI/Build] Fix Args for _get_logits_warper in Sampler Test vllm-project/vllm#5922

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pass device in Logits Processor's init #29804

Pass device in Logits Processor's init #29804

zucchini-nlp commented Mar 22, 2024

HuggingFaceDocBuilderDev commented Mar 22, 2024

gante left a comment

zucchini-nlp commented Mar 25, 2024

zucchini-nlp commented Apr 22, 2024

zucchini-nlp commented May 9, 2024

gante left a comment

zucchini-nlp commented May 22, 2024 •

edited

Loading

ArthurZucker commented May 23, 2024

ArthurZucker left a comment

ArthurZucker May 23, 2024

zucchini-nlp May 24, 2024

ArthurZucker May 24, 2024

ArthurZucker May 24, 2024

zucchini-nlp May 24, 2024

ArthurZucker May 23, 2024

zucchini-nlp May 24, 2024

ArthurZucker May 23, 2024

ArthurZucker May 23, 2024

zucchini-nlp May 24, 2024

ArthurZucker May 24, 2024

ArthurZucker left a comment

zucchini-nlp commented Jun 4, 2024

Pass device in Logits Processor's init #29804

Pass device in Logits Processor's init #29804

Conversation

zucchini-nlp commented Mar 22, 2024

What does this PR do?

Who can review?

HuggingFaceDocBuilderDev commented Mar 22, 2024

gante left a comment

Choose a reason for hiding this comment

zucchini-nlp commented Mar 25, 2024

zucchini-nlp commented Apr 22, 2024

zucchini-nlp commented May 9, 2024

gante left a comment

Choose a reason for hiding this comment

zucchini-nlp commented May 22, 2024 • edited Loading

ArthurZucker commented May 23, 2024

ArthurZucker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArthurZucker left a comment

Choose a reason for hiding this comment

zucchini-nlp commented Jun 4, 2024

zucchini-nlp commented May 22, 2024 •

edited

Loading