Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Offloaded KV Cache #31325

Merged
merged 8 commits into from
Aug 1, 2024
Merged

Offloaded KV Cache #31325

merged 8 commits into from
Aug 1, 2024

Commits on Jul 23, 2024

  1. Initial implementation of OffloadedCache

    @n17s
    n17s committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    8e57b08 View commit details
    Browse the repository at this point in the history

  2. enable usage via cache_implementation

    @n17s
    n17s committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    d0e8666 View commit details
    Browse the repository at this point in the history

  3. Address feedback, add tests, remove legacy methods.

    @n17s
    n17s committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    2e63564 View commit details
    Browse the repository at this point in the history

  4. Remove flash-attn, discover synchronization bugs, fix bugs

    @n17s
    n17s committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    cf31e0e View commit details
    Browse the repository at this point in the history

  5. Prevent usage in CPU only mode

    @n17s
    n17s committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    daf8702 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Add a section about offloaded KV cache to the docs

    @n17s
    n17s committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    4795032 View commit details
    Browse the repository at this point in the history

  2. Fix typos in docs

    @n17s
    n17s committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1a76762 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. Clarifications and better explanation of streams

    @n17s
    n17s committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    667811a View commit details
    Browse the repository at this point in the history