-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fsspec.exceptions.FSTimeoutError when downloading dataset #7164
Comments
Hi ! If you check the dataset loading script here you'll see that it downloads the data from OpenSLR, and apparently their storage has timeout issues. It would be great to ultimately host the dataset on Hugging Face instead. In the meantime I can only recommend to try again later :/ |
Ok, still many thanks! |
I'm also getting this same error but for |
in v3 we cleaned the download parts of the library to make it more robust for HF downloads and to simplify support of script-based datasets. As a side effect it's not the same code that is used for other hosts, maybe time out handling changed. Anyway it should be possible to tweak fsspec to use retries For example using aiohttp_retry maybe (haven't tried) ? import fsspec
from aiohttp_retry import RetryClient
fsspec.filesystem("http")._session = RetryClient() related topic : #7175 |
Adding a timeout argument to the fs.get_file(path, temp_file.name, callback=callback, timeout=3600) Setting This is using Edit: This doesn't seem to change the timeout time, but add a second timeout counter (probably in Edit 2: TLDR; This fixes it: import datasets, aiohttp
dataset = datasets.load_dataset(
dataset_name,
storage_options={'client_kwargs': {'timeout': aiohttp.ClientTimeout(total=3600)}}
) |
Describe the bug
I am trying to download the
librispeech_asr
clean
dataset, which results in aFSTimeoutError
exception after downloading around 61% of the data.Steps to reproduce the bug
The output is as follows:
Expected behavior
Complete the download
Environment info
Python version 3.12.6
Dependencies:
MacOS 14.6.1 (23G93)
The text was updated successfully, but these errors were encountered: