Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Train] Allow local datasets in HuggingFaceTrainer #34485

Merged
merged 2 commits into from
Apr 18, 2023
Merged

[Train] Allow local datasets in HuggingFaceTrainer #34485

merged 2 commits into from
Apr 18, 2023

Conversation

Yard1
Copy link
Member

@Yard1 Yard1 commented Apr 17, 2023

Why are these changes needed?

Loosens restrictions in HuggingFaceTrainer to make the datasets argument optional and thus allow Hugging Face Datasets to be used directly.

Related issue number

Closes #34476 (comment)

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@Yard1 Yard1 requested a review from gjoliver April 17, 2023 20:11
gjoliver
gjoliver approved these changes Apr 17, 2023
View reviewed changes
Copy link
Member

@gjoliver gjoliver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, sick!! just 1 nit.
thanks for being so quick.

python/ray/train/huggingface/huggingface_trainer.py Outdated Show resolved Hide resolved
@Yard1 Yard1 changed the title Allow local datasets in HuggingFaceTrainer [Train] Allow local datasets in HuggingFaceTrainer Apr 17, 2023
@Yard1
Clarify
56289d6 
Signed-off-by: Antoni Baum <[email protected]>
@Yard1 Yard1 merged commit d010563 into ray-project:master Apr 18, 2023
@Yard1 Yard1 deleted the hf_trainer_allow_non_ray_datasets branch April 18, 2023 20:29
elliottower pushed a commit to elliottower/ray that referenced this pull request Apr 22, 2023
@Yard1 @elliottower
[Train] Allow local datasets in HuggingFaceTrainer (ray-project#34485)
00d7b9b 
* Allow local datasets in HuggingFaceTrainer

Signed-off-by: Antoni Baum <[email protected]>

* Clarify

Signed-off-by: Antoni Baum <[email protected]>

---------

Signed-off-by: Antoni Baum <[email protected]>
Signed-off-by: elliottower <[email protected]>
ProjectsByJackHe pushed a commit to ProjectsByJackHe/ray that referenced this pull request May 4, 2023
@Yard1 @ProjectsByJackHe
[Train] Allow local datasets in HuggingFaceTrainer (ray-project#34485)
8fd6ccf 
* Allow local datasets in HuggingFaceTrainer

Signed-off-by: Antoni Baum <[email protected]>

* Clarify

Signed-off-by: Antoni Baum <[email protected]>

---------

Signed-off-by: Antoni Baum <[email protected]>
Signed-off-by: Jack He <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gjoliver gjoliver gjoliver approved these changes

Assignees

@gjoliver gjoliver

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[AIR] Make it possible for HuggingFaceTrainer to take HF datasets directly
2 participants
@Yard1 @gjoliver