Update datasets-download-stats.md #1466
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
davanstrien
approved these changes
Oct 22, 2024
Co-authored-by: Daniel van Strien <[email protected]>
Co-authored-by: Daniel van Strien <[email protected]>
|
merging this one for now, we can still add more details later if needed |
julien-c
reviewed
Oct 24, 2024
|
|
||
| * The download count is the same regardless of whether the data is directly stored on the Hub repo or if the repository has a [script](/docs/datasets/dataset_script) to load the data from an external source. | ||
| * If a user manually downloads the data using tools like `wget` or the Hub's user interface (UI), those downloads will not be included in the download count. | ||
| ## Before Setpember 2024 |
julien-c
reviewed
Oct 24, 2024
| @@ -2,7 +2,11 @@ | |||
|
|
|||
| ## How are download stats generated for datasets? | |||
There was a problem hiding this comment.
would be clearer maybe to word it like: "How are downloads counted for datasets" (same for models)
julien-c
reviewed
Oct 24, 2024
| @@ -2,7 +2,11 @@ | |||
|
|
|||
| ## How are download stats generated for datasets? | |||
|
|
|||
| The Hub provides download stats for all datasets loadable via the `datasets` library. To determine the number of downloads, the Hub counts every time `load_dataset` is called in Python, excluding Hugging Face's CI tooling on GitHub. No information is sent from the user, and no additional calls are made for this. The count is done server-side as we serve files for downloads. This means that: | |||
| Counting the number of downloads for datasets is not a trivial task, as a single dataset repository might contain multiple files, from multiple subsets and splits (e.g. train/validation/test) and sometimes with many files in a single split. To solve this issue and avoid counting one person's download multiple times, we treat all files downloaded by a user within a 5-minute window as a single dataset download. This counting happens automatically on our servers when files are downloaded (through GET or HEAD requests), with no need to collect any user information or make additional calls. | |||
There was a problem hiding this comment.
by a user (based on their IP address)
maybe add this to make it clearer for us? @lhoestq
|
sounds good ! opened #1469 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updated the download count method, and I kept how it was working before september 2024 (since odler data an be viewer from Enterprise analytics) cc @julien-c