Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HF dataset loading optimizations #623

Merged
merged 11 commits into from
Jun 14, 2024
Merged

HF dataset loading optimizations #623

merged 11 commits into from
Jun 14, 2024

Commits on Jun 13, 2024

  1. Add util methods for loading and saving to disk

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    f4dbde6 View commit details
    Browse the repository at this point in the history

  2. Add logic for loading dataset using disk cache

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    82d0e9d View commit details
    Browse the repository at this point in the history

  3. Pass cache dir into downstream evals

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    b4acd09 View commit details
    Browse the repository at this point in the history

  4. Pass cache dir using training config

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    561d642 View commit details
    Browse the repository at this point in the history

  5. Simplify, robustify but slightly unoptimize load_dataset logic

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9f2d9d2 View commit details
    Browse the repository at this point in the history

  6. Add type hints for load_dataset

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    c55974d View commit details
    Browse the repository at this point in the history

  7. Remove type ignore for downstream evaluator construction

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    97911fd View commit details
    Browse the repository at this point in the history

  8. Re-add type ignore

    @2015aroras
    2015aroras committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    f934bb7 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Move load_hf_dataset to utils

    @2015aroras
    2015aroras committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    10737a2 View commit details
    Browse the repository at this point in the history

  2. Add load_hf_dataset tests

    @2015aroras
    2015aroras committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    2058edd View commit details
    Browse the repository at this point in the history

  3. Update CHANGELOG

    @2015aroras
    2015aroras committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    b5bd9ff View commit details
    Browse the repository at this point in the history