Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FSDP2][NF4Tensor][2/n] implement torch.chunk and other ops #150

Merged
merged 47 commits into from
May 1, 2024
Merged

[FSDP2][NF4Tensor][2/n] implement torch.chunk and other ops #150

merged 47 commits into from
May 1, 2024

Commits on Apr 4, 2024

  1. proof of concept for FSDP2 + NF4Tensor 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    0a13e6a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into main

    @cpuhrsch
    cpuhrsch authored Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9a56eaa View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. fsdp extention for tensor subclass 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 11, 2024
    Configuration menu
    Copy the full SHA
    8180540 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. support fp32 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    95b03e1 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Merge branch 'pytorch-labs:main' into main

    @weifengpy
    weifengpy authored Apr 16, 2024
    Configuration menu
    Copy the full SHA
    3ac9d81 View commit details
    Browse the repository at this point in the history

  2. UNIT TEST FOR STATE DICT 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    38461b3 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2024

  1. implement to 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    bc7a764 View commit details
    Browse the repository at this point in the history

  2. remove torch.override from torch function 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    8b1d037 View commit details
    Browse the repository at this point in the history

  3. use dtype in compile unit test 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    7ff6855 View commit details
    Browse the repository at this point in the history

  4. add dtype in all unit test 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    d9bcf71 View commit details
    Browse the repository at this point in the history

  5. keep original dtype 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    923bef2 View commit details
    Browse the repository at this point in the history

  6. fix linter 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    e15d244 View commit details
    Browse the repository at this point in the history

  7. use torch testing @parametrize 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    d4beb8f View commit details
    Browse the repository at this point in the history

  8. remove unused import 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    f41cb3d View commit details
    Browse the repository at this point in the history

  9. Merge branch 'pytorch-labs:main' into main

    @weifengpy
    weifengpy authored Apr 17, 2024
    Configuration menu
    Copy the full SHA
    952fbdd View commit details
    Browse the repository at this point in the history

  10. sm8 for fp16 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    950d9fd View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2024

  1. remove sm check for fp16 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    d4eae0b View commit details
    Browse the repository at this point in the history

  2. skip 2.2.2 and below for tracing tensor subclass 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    9444f2c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'pytorch-labs:main' into main

    @weifengpy
    weifengpy authored Apr 18, 2024
    Configuration menu
    Copy the full SHA
    b2c3c02 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. include kwargs 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    9be2de3 View commit details
    Browse the repository at this point in the history

  2. raise unimplemented 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    2981393 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into main

    @weifengpy
    weifengpy authored Apr 19, 2024
    Configuration menu
    Copy the full SHA
    3ced998 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'pytorch-labs:main' into main

    @weifengpy
    weifengpy authored Apr 19, 2024
    Configuration menu
    Copy the full SHA
    3f1e19a View commit details
    Browse the repository at this point in the history

  5. fsdp2 ops 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    761416a View commit details
    Browse the repository at this point in the history

  6. better diff layout 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    c656f1e View commit details
    Browse the repository at this point in the history

  7. set pg size in metadata 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    c56d7e2 View commit details
    Browse the repository at this point in the history

  8. remove irrelevant changes 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    d656b93 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2024

  1. add unit test 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 20, 2024
    Configuration menu
    Copy the full SHA
    5c4fe2b View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Merge branch 'main' into main

    @msaroufim
    msaroufim authored Apr 26, 2024
    Configuration menu
    Copy the full SHA
    613bf67 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. torch.chunk and cpu offloading ops 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    3933bfa View commit details
    Browse the repository at this point in the history

  2. remove strict same metadata check 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    9e6b4ec View commit details
    Browse the repository at this point in the history

  3. skip tests that needs cuda 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    857b8db View commit details
    Browse the repository at this point in the history

  4. use /( in regex match 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    8e3de02 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. fix regex 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    912998b View commit details
    Browse the repository at this point in the history

  2. skip tests if no cuda 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    8926ee1 View commit details
    Browse the repository at this point in the history

  3. skip unit test if no cuda 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 28, 2024
    Configuration menu
    Copy the full SHA
    6f834ce View commit details
    Browse the repository at this point in the history

  4. Merge branch 'pytorch:main' into main

    @weifengpy
    weifengpy authored Apr 28, 2024
    Configuration menu
    Copy the full SHA
    a8a5aaa View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. assert cpu device 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    699079d View commit details
    Browse the repository at this point in the history

  2. name args[0] as nf4tensor 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    c8b047c View commit details
    Browse the repository at this point in the history

  3. utils for apply to inner tensors and constructor 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    925602c View commit details
    Browse the repository at this point in the history

  4. use original copy_ 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    e36ab6c View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. decorator for args check 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed May 1, 2024
    Configuration menu
    Copy the full SHA
    a007027 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into main

    @cpuhrsch
    cpuhrsch authored May 1, 2024
    Configuration menu
    Copy the full SHA
    c352552 View commit details
    Browse the repository at this point in the history

  3. INNER_TENSOR_NAMES_FOR_SHARDING and unify assert in split and new_zeros 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed May 1, 2024
    Configuration menu
    Copy the full SHA
    c83fdad View commit details
    Browse the repository at this point in the history

  4. Merge branch 'pytorch:main' into main

    @weifengpy
    weifengpy authored May 1, 2024
    Configuration menu
    Copy the full SHA
    574fecd View commit details
    Browse the repository at this point in the history

  5. indicate private constant with _ 

    Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
    @weifengpy
    weifengpy committed May 1, 2024
    Configuration menu
    Copy the full SHA
    f27760b View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into fsdp2ops

    @weifengpy
    weifengpy authored May 1, 2024
    Configuration menu
    Copy the full SHA
    b4f51b9 View commit details
    Browse the repository at this point in the history