Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix compatibility of adapters with HF Accelerate auto device-mapping #678

Merged
merged 2 commits into from
Apr 22, 2024

Conversation

calpt
Copy link
Member

@calpt calpt commented Apr 14, 2024

Adapters currently does not work correctly with passing device_map="auto" in a model's from_pretrained(). Device auto-mapping is handled by HF accelerate, which wraps the original module forward method.

This PR fixes compatibility of Adapters' post-hoc model wrapping with Accelerate's device auto-mapping via wrapping the forward pass.

Fixing this is required for enabling quantized training of adapters (bottleneck & prefix-tuning) in #663.

The fix can be tested like this:

import adapters
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "t5-small"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    device_map="auto"
)

adapters.init(model)
model.add_adapter("my_adapter")
model.train_adapter("my_adapter")

model.to("cuda")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
inputs = inputs.to("cuda")

is_accessed = False

def hook_fn(module, args, output):
    global is_accessed
    is_accessed = True
    return output

adapter = model.get_adapter("my_adapter")
first_layer_module = adapter[0]["output_adapter"]
first_layer_module.register_forward_hook(hook_fn)

outputs = model(**inputs)
print(outputs)

assert is_accessed

@calpt calpt added the bug Something isn't working label Apr 14, 2024
@calpt calpt changed the title Fix compatibility of adapters with HF Accelerate auto device-map Fix compatibility of adapters with HF Accelerate auto device-mapping Apr 14, 2024
Copy link
Member

@hSterz hSterz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@calpt calpt merged commit 233db31 into adapter-hub:main Apr 22, 2024
3 checks passed
@calpt calpt deleted the fix/auto_device_map branch April 22, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants