New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug about weight sharing in AutoFormer #232

Open

variant-star opened this issue Mar 28, 2024 · 0 comments

variant-star commented Mar 28, 2024 •

edited

Loading

Cream/AutoFormer/model/module/qkv_super.py

Lines 72 to 77 in 4a13c40

    
           def sample_weight(weight, sample_in_dim, sample_out_dim): 
        
               sample_weight = weight[:, :sample_in_dim] 
        
               sample_weight = torch.cat([sample_weight[i:sample_out_dim:3, :] for i in range(3)], dim =0) 
        
               return sample_weight

I think, there's something wrong in the way weight sharing is done here. I think this code should be:

    N = weight.size(0) // 3
    sample_weight = torch.cat([sample_weight[i*N:i*N+sample_out_dim//3, :] for i in range(3)], dim=0)

To be more intuitive, I drew a schematic diagram to represent the way 4 and 5 heads SA is shared with Linear.weight.

Maybe I misunderstood the implementation here, can you help check it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment