Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fs.http: prevent hangs under some network conditions #7460

Merged
merged 2 commits into from
Mar 28, 2022
Merged

fs.http: prevent hangs under some network conditions #7460

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1574d9a
fs.http: force cleanup of closed connections
dtrifiro Mar 14, 2022
12de7e7
fs.http: set sock_read timeout
dtrifiro Mar 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions dvc/fs/http.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,11 +81,20 @@ def _prepare_credentials(self, **config):
f"Auth method {auth_method!r} is not supported."
)

# Force cleanup of closed SSL transports.
# https://github.com/iterative/dvc/issues/7414
connector_kwargs = {"enable_cleanup_closed": True}

if "ssl_verify" in config:
with fsspec_loop():
client_kwargs["connector"] = aiohttp.TCPConnector(
ssl=make_context(config["ssl_verify"])
)
connector_kwargs.update(ssl=make_context(config["ssl_verify"]))

with fsspec_loop():
client_kwargs["connector"] = aiohttp.TCPConnector(
**connector_kwargs
)
# The connector should not be owned by aiohttp.ClientSession since
# it is closed by fsspec (HTTPFileSystem.close_session)
client_kwargs["connector_owner"] = False

# Allow reading proxy configurations from the environment.
client_kwargs["trust_env"] = True
Expand Down Expand Up @@ -114,7 +123,7 @@ async def get_client(self, **kwargs):
total=None,
connect=self.REQUEST_TIMEOUT,
sock_connect=self.REQUEST_TIMEOUT,
sock_read=None,
sock_read=self.REQUEST_TIMEOUT,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming that setting the sock_read=self.REQUEST_TIMEOUT (60s) will not create any issues: the default buffer size is 2**16,

See ClientSession._request:
https://github.com/aio-libs/aiohttp/blob/f5ff95efe278c470a2ff65cabbb5f5f08ba07416/aiohttp/client.py#L511-L519

https://github.com/aio-libs/aiohttp/blob/f5ff95efe278c470a2ff65cabbb5f5f08ba07416/aiohttp/client_proto.py#L17

https://github.com/aio-libs/aiohttp/blob/f5ff95efe278c470a2ff65cabbb5f5f08ba07416/aiohttp/streams.py#L111

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious to see if only this will solve the problem in dvc-bench, and a bit hesitant on adding TCPConnector
without trying out this solution first. Would you mind opening another PR with just this line change? We can merge that, and we can keep this PR open for a 1-2 days to monitor.

Copy link
Contributor Author

@dtrifiro dtrifiro Mar 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried, it solves the issues on my machine ™️ (MacOS) but it still freezes on the dvc-bench CI

)

return RetryClient(**kwargs)
Expand Down