feat(core): service add HuggingFace file system #3670

morristai · 2023-11-27T07:36:01Z

Description

close: #3605

Addition Note

HuggingFace doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the HuggingFace Hub's source code.
For message.rs, I'm unsure whether we should drop unused fields or add #[allow(dead_code)] to disable warnings.

core/src/services/huggingface/backend.rs

core/src/services/huggingface/core.rs

core/src/services/huggingface/message.rs

core/src/services/huggingface/mod.rs

Wauplin

Hey there! I'm @Wauplin, maintainer of huggingface_hub -the Python client library for the Hugging Face Hub-. This is a very cool integration! I quickly had a look and added a few high-level comments about the existing huggingface ecosystem (to better align the naming conventions).

I also wanted to point you to https://github.com/huggingface/hf-hub which is the rust client to interact with the Hub. Though it's not as complete as the Python equivalent, it already has support for most file read/write use cases -including compatibility with the HF cache system- . It would be good to check what's there to avoid duplicating logic across libraries. cc @Narsil who worked on this

core/src/services/huggingface/backend.rs

core/src/services/huggingface/mod.rs

core/src/types/scheme.rs

core/src/services/huggingface/docs.md

Xuanwo · 2023-11-28T14:34:57Z

I also wanted to point you to https://github.com/huggingface/hf-hub which is the rust client to interact with the Hub. Though it's not as complete as the Python equivalent, it already has support for most file read/write use cases -including compatibility with the HF cache system- .

Thank you for the advice, hf-hub is an excellent project! However, OpenDAL prefers to implement all APIs using its own HttpClient to ensure full control, eliminating the need to interact with various SDKs and keep it's deps tree as low as possible.

Wauplin · 2023-11-28T14:40:58Z

Thank you for the advice, hf-hub is an excellent project! However, OpenDAL prefers to implement all APIs using its own HttpClient to ensure full control, eliminating the need to interact with various SDKs and keep it's deps tree as low as possible.

Fair enough @Xuanwo! I just wanted to make sure you know about it :)

Xuanwo · 2023-11-28T14:43:27Z

I just wanted to make sure you know about it :)

You are so nice! And thanks for the review. ❤️

core/src/services/huggingface/backend.rs

core/src/services/huggingface/core.rs

core/src/services/huggingface/lister.rs

core/src/types/scheme.rs

morristai · 2023-11-30T05:44:41Z

Seems not fixed. With ConfigDeserializer, we don't need to use write map.get("abc") by hand.

(It appears that I am unable to reply in that review block, so I'll respond here.)
@Xuanwo, Sorry, it's my negligence. But I want to point out that I'm referring to service Etcd. It seems like it's also miss refactored?
https://github.com/apache/incubator-opendal/blob/0711e6ce97660380628b72a010d092fd29ec9553/core/src/services/etcd/backend.rs#L200

Xuanwo · 2023-11-30T06:26:24Z

Sorry, it's my negligence. But I want to point out that I'm referring to service Etcd. It seems like it's also miss refactored?

Yes, there is an on-going effort to refactor them one by one: #3240

morristai · 2023-12-01T09:29:17Z

Yes, there is an on-going effort to refactor them one by one: #3240

I believe there might be a misunderstanding. After checking the refactoring progress list and the pull request code, it appears that Etcd has already been refactored, but the from_map part seems to have been overlooked. Could you please verify this from your end? I'm not certain if I've missed something here. (By the way, I am currently on a business trip, so my response may be delayed.)

Xuanwo · 2023-12-01T10:11:47Z

I believe there might be a misunderstanding. After checking the refactoring progress list and the pull request code, it appears that Etcd has already been refactored, but the from_map part seems to have been overlooked. Could you please verify this from your end?

Seems etcd missed this part, we need to fix it, tracked at #3702

By the way, I am currently on a business trip, so my response may be delayed

Enjoy your trip!

Xuanwo

Thanks!

Xuanwo · 2023-12-02T10:50:12Z

I have setup a repo at https://huggingface.co/datasets/opendal/huggingface-testdata

Would you like to add a behaivor test for this?

morristai · 2023-12-02T13:30:02Z

I have setup a repo at https://huggingface.co/datasets/opendal/huggingface-testdata

Would you like to add a behaivor test for this?

Yes, I'd love to work on that.

morristai added 2 commits November 27, 2023 02:28

feat: service add huggingface file system

42a51e7

chore: fix typo

0ea42a3

morristai requested review from Xuanwo and PsiACE as code owners November 27, 2023 07:36

github-actions bot added the releases-note/feat label Nov 27, 2023

Xuanwo reviewed Nov 27, 2023

View reviewed changes

Wauplin reviewed Nov 28, 2023

View reviewed changes

core/src/services/huggingface/backend.rs Show resolved Hide resolved

core/src/services/huggingface/mod.rs Outdated Show resolved Hide resolved

core/src/types/scheme.rs Outdated Show resolved Hide resolved

core/src/services/huggingface/docs.md Outdated Show resolved Hide resolved

morristai added 11 commits November 28, 2023 19:51

feat: refactor read_token with token

28018d6

feat: add HuggingFaceConfig implementation

94d6725

chore: use better syntax implementation

62714fc

chore: use format_authorization_by_bearer instead

1c027d9

chore: use Eq, PartialEq to make testing cleaner

fab254e

feat: change scheme from huggingface to hf

578ad03

feat: rename HuggingFace to Huggingface

272a4a7

chore: move message.rs to core.rs

5d0389f

feat: drop dependency serde_urlencoded

66cb6ab

chore: fix for cargo clippy

444388d

Merge branch 'main' into feat/huggingface_filesystem_support

5d7ab2f

Xuanwo reviewed Nov 30, 2023

View reviewed changes

Xuanwo mentioned this pull request Dec 1, 2023

etcd implement from_map wrongly #3702

Closed

morristai added 3 commits December 2, 2023 05:05

feat: rename huggingface scheme

71f3a71

chore: polish code

3d9ac93

chore: use http::header for well-known header names

75a0d89

morristai added 4 commits December 2, 2023 05:11

feat: fix for PR review

58e92ae

Merge branch 'main' into feat/huggingface_filesystem_support

0526269

chore: read hf_read to hf_resolve

736845b

chore: read hf_read to hf_resolve

3194a04

Xuanwo approved these changes Dec 2, 2023

View reviewed changes

Xuanwo merged commit 547c23d into apache:main Dec 2, 2023
171 checks passed

morristai mentioned this pull request Dec 3, 2023

Add behavior test for Hugging Face File System #3708

Closed

morristai deleted the feat/huggingface_filesystem_support branch December 5, 2023 00:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): service add HuggingFace file system #3670

feat(core): service add HuggingFace file system #3670

morristai commented Nov 27, 2023

Wauplin left a comment

Xuanwo commented Nov 28, 2023

Wauplin commented Nov 28, 2023

Xuanwo commented Nov 28, 2023

morristai commented Nov 30, 2023

Xuanwo commented Nov 30, 2023

morristai commented Dec 1, 2023

Xuanwo commented Dec 1, 2023 •

edited

Loading

Xuanwo left a comment

Xuanwo commented Dec 2, 2023

morristai commented Dec 2, 2023

feat(core): service add HuggingFace file system #3670

feat(core): service add HuggingFace file system #3670

Conversation

morristai commented Nov 27, 2023

Description

Addition Note

Wauplin left a comment

Choose a reason for hiding this comment

Xuanwo commented Nov 28, 2023

Wauplin commented Nov 28, 2023

Xuanwo commented Nov 28, 2023

morristai commented Nov 30, 2023

Xuanwo commented Nov 30, 2023

morristai commented Dec 1, 2023

Xuanwo commented Dec 1, 2023 • edited Loading

Xuanwo left a comment

Choose a reason for hiding this comment

Xuanwo commented Dec 2, 2023

morristai commented Dec 2, 2023

Xuanwo commented Dec 1, 2023 •

edited

Loading