Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add FusedRMSNorm (Triton kernel, +15% eager), Add NPLayerNorm, Enable config selectable Norm Type #181

Merged
merged 31 commits into from
Apr 5, 2024
Merged

Add FusedRMSNorm (Triton kernel, +15% eager), Add NPLayerNorm, Enable config selectable Norm Type #181

merged 31 commits into from
Apr 5, 2024

Commits on Mar 29, 2024

  1. Create fused_rms_norm.py

    @lessw2020
    lessw2020 authored Mar 29, 2024
    Configuration menu
    Copy the full SHA
    457abee View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2024

  1. start unifying all norms with single base class

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    a2435df View commit details
    Browse the repository at this point in the history

  2. add strEnum for NormTypes

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    de2a41b View commit details
    Browse the repository at this point in the history

  3. link norm type control to model config from train config

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    e8782f1 View commit details
    Browse the repository at this point in the history

  4. link norm type control to model config from train config

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    d797d27 View commit details
    Browse the repository at this point in the history

  5. all working

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    81f3402 View commit details
    Browse the repository at this point in the history

  6. linting

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    a1b45ba View commit details
    Browse the repository at this point in the history

  7. linting, remove triton file from linting check

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    0c0d41f View commit details
    Browse the repository at this point in the history

  8. add default rms in config_manager

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    47a5791 View commit details
    Browse the repository at this point in the history

  9. lint default rms in config_manager, update 7b, 70b toml

    @lessw2020
    lessw2020 committed Mar 30, 2024
    Configuration menu
    Copy the full SHA
    be88137 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2024

  1. remove alternative backward pass section (saved in kernel repo)

    @lessw2020
    lessw2020 committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    ff37024 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. revert reshard_after_forward from MFU tuning

    @lessw2020
    lessw2020 committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    7c0c62b View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2024

  1. merge all norms into single file per PR feedback

    @lessw2020
    lessw2020 committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    f81ff3f View commit details
    Browse the repository at this point in the history

  2. move norms.py into models dir

    @lessw2020
    lessw2020 committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    87def6a View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. remove StrEnum, add localhost:0

    @lessw2020
    lessw2020 committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    df29e6e View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. change reshape to view in triton code

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    aef4e0c View commit details
    Browse the repository at this point in the history

  2. standardize norms to 4 express names

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    9088524 View commit details
    Browse the repository at this point in the history

  3. remove normbase

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    d32fe71 View commit details
    Browse the repository at this point in the history

  4. correct merge conflict in run.sh

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    da4bd20 View commit details
    Browse the repository at this point in the history

  5. update to remove debug2d.toml

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    310be84 View commit details
    Browse the repository at this point in the history

  6. correct mergeconflict 2 in run.sh

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    6747119 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into fused_rms_norm

    @lessw2020
    lessw2020 authored Apr 5, 2024
    Configuration menu
    Copy the full SHA
    2d46786 View commit details
    Browse the repository at this point in the history

  8. linting

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    83ec4b6 View commit details
    Browse the repository at this point in the history

  9. add comment regarding flake8 N803,N806

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    8750043 View commit details
    Browse the repository at this point in the history

  10. add check for fused_rmsnorm when TP active.

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    701e0d3 View commit details
    Browse the repository at this point in the history

  11. linting

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    2a3ac78 View commit details
    Browse the repository at this point in the history

  12. formatting

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    d5b4b32 View commit details
    Browse the repository at this point in the history

  13. remove init_weights, streamline reset_params

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    5a937e9 View commit details
    Browse the repository at this point in the history

  14. updated TP norm msg, reset 7b steps

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    54cf135 View commit details
    Browse the repository at this point in the history

  15. update kernel name to TritonFusedRMSNorm

    @lessw2020
    lessw2020 committed Apr 5, 2024
    Configuration menu
    Copy the full SHA
    239b71f View commit details
    Browse the repository at this point in the history

  16. Merge branch 'main' into fused_rms_norm

    @lessw2020
    lessw2020 authored Apr 5, 2024
    Configuration menu
    Copy the full SHA
    1d84539 View commit details
    Browse the repository at this point in the history