Use force_close and concurrency restriction #3733

bmbouter · 2018-10-31T21:46:37Z

Use force_close and concurrency restriction

The use of force_close was inspired by:

aio-libs/aiohttp#2867 (comment)

Causes Factory and Downloaders to use force_close which causes the TCP
connection to close with each request.
Ups the default concurrency to 20
Moves the concurrency restriction feature to the Downloaders and
updates HttpDownloader and FileDownloader to use it.
Stops using aiohttp for concurrency restriction because it's done in
the downloaders.
The Factory now configures the concurrency restriction using the value
on the remote.
creates a _run() method that subclassed Downloaders now need to
implement instead of run().

https://pulp.plan.io/issues/4036
https://pulp.plan.io/issues/4075

closes #4036
closes #4075

pep8speaks · 2018-10-31T21:46:41Z

Hello @bmbouter! Thanks for submitting the PR.

There are no PEP8 issues in the file plugin/pulpcore/plugin/download/base.py !
There are no PEP8 issues in the file plugin/pulpcore/plugin/download/factory.py !
There are no PEP8 issues in the file plugin/pulpcore/plugin/download/file.py !
There are no PEP8 issues in the file plugin/pulpcore/plugin/download/http.py !
There are no PEP8 issues in the file pulpcore/pulpcore/app/models/repository.py !

dralley · 2018-10-31T22:40:08Z

You mention "force_full" by accident in the commit message but nbd

dralley · 2018-10-31T22:41:36Z

plugin/pulpcore/plugin/download/base.py

+        if semaphore:
+            self.semaphore = semaphore
+        else:
+            self.semaphore = asyncio.Semaphore()  # This will always be acquired


This could be simplified to self.semaphore = semaphore or asyncio.Semaphore() but this way might be more clear to readers, up to you.

I was trying to remember that idiom! Thanks @dralley

I like your simplified idiom personally :)

Actually I'm going to keep it as is because even though I like the shortened idiom, this for lets me keep the comment which I think is helpful since creating a random Semaphore might not be obvious why.

dralley · 2018-10-31T22:45:51Z

Without having tested it, LGTM. I'll let someone else review + test though since I don't want to tear up my environment.

dralley · 2018-10-31T22:47:10Z

Do we have a way to regression-test this? Would it make sense to do so? Perhaps we could write a test and just have it be skipped by default, if it would take too long to run every time?

codecov · 2018-10-31T22:58:02Z

Codecov Report

Merging #3733 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #3733   +/-   ##
======================================
  Coverage    54.9%   54.9%           
======================================
  Files          64      64           
  Lines        2781    2781           
======================================
  Hits         1527    1527           
  Misses       1254    1254

Impacted Files	Coverage Δ
pulpcore/pulpcore/app/models/repository.py	`54.86% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a462fec...adec0fb. Read the comment docs.

bmbouter · 2018-11-01T13:41:55Z

@dralley when you say regression test do you mean a pulp smash test? Also this PR has several apsects, so we need to think about testing each one at some point. Overall, one option to do that with would be to make a dummy aiohttp server that would serve HTTP 1.1 with session continuation back to Pulp's downloaders at test time and then verify that the client disconnects even though the server tells it to keep going. I'm going to file two Pulp Test issues to have those functional tests tracked.

bmbouter · 2018-11-01T13:45:34Z

Now that concurrency restriction is in the downloaders themselves, I believe connection_limit doesn't make sense as the name anymore. Specifically it doesn't make sense with the FileDownloader which uses the option, but doesn't even have a "connection". Since this PR introduces this issue, we should resolve it with this PR by renaming s/connection_limit/concurrency_limit/. I will do that now.

bmbouter · 2018-11-01T14:24:14Z

Actually rather than including the name change with this PR I'm going to make a followon issue. That will give us a chance to pick the best name together before we make the change. I'll file an issue today.

dkliban · 2018-11-01T14:40:11Z

plugin/pulpcore/plugin/download/http.py

@@ -54,6 +54,10 @@ class HttpDownloader(BaseDownloader):
    allow for an active download to be arbitrarily long, while still detecting dead or closed
    sessions even when TCPKeepAlive is disabled.

+    If a session is not provided, the one created will disable HTTP session force TCP closure with


s/will disable HTTP session force TCP closure with each request/will force TCP connection closure after each request/

fixed in next push, ty

bmbouter · 2018-11-01T14:49:16Z

download concurrency rename ticket is here: https://pulp.plan.io/issues/4114

bmbouter · 2018-11-01T16:27:57Z

Here are the two testing tickets I wrote up:
https://pulp.plan.io/issues/4114
https://pulp.plan.io/issues/4115

dkliban

Thank you!

gmbnomis

I think there is one problem with the current implementation. Otherwise, LGTM.

gmbnomis · 2018-11-01T15:10:55Z

plugin/pulpcore/plugin/download/http.py

@@ -158,7 +163,8 @@ def __init__(self, url, session=None, auth=None, proxy=None, proxy_auth=None,
        return DownloadResult(path=self.path, artifact_attributes=self.artifact_attributes,
                              url=self.url)

-    @backoff.on_exception(backoff.expo, aiohttp.ClientResponseError, max_tries=10, giveup=giveup)
+    @backoff.on_exception(backoff.expo, aiohttp.ClientResponseError,


Not sure, but doesn't this essentially have the same behavior as before? The backoff and waiting time is implemented as a decorator, i.e. happens outside of the method when the semaphore is not acquired.

I think the semaphore must be acquired outside of the run function. My idea would be to implement the limiting mechanism in the base class:

async def run(self, extra_data=None): async with self.semaphore: return await self._run(extra_data)

And the actual downloader would have:

@backoff.on_exception(backoff.expo, aiohttp.ClientResponseError, max_tries=10, giveup=http_giveup) async def _run(self, extra_data=None): ...

just as before.

This has the additional benefit that the actual Downloader does not need to re-implement the limiting mechanism.

@gmbnomis indeed! Thank you again for your astute observation. I'm going to rework this now, and I'm going to file a test to ensure we're testing backoff correctly in core's functional tests.

I filed this test issue so that we would ensure this with each pulp_smash run on core. https://pulp.plan.io/issues/4118

The use of force_close was inspired by: aio-libs/aiohttp#2867 (comment) - Causes Factory and Downloaders to use force_close which causes the TCP connection to close with each request. - Ups the default concurrency to 20 - Moves the concurrency restriction feature to the Downloaders and updates HttpDownloader and FileDownloader to use it. - Stops using aiohttp for concurrency restriction because it's done in the downloaders. - The Factory now configures the concurrency restriction using the value on the remote. - creates a _run() method that subclassed Downloaders now need to implement instead of run(). https://pulp.plan.io/issues/4036 https://pulp.plan.io/issues/4075 closes #4036 closes #4075

bmbouter · 2018-11-01T19:51:36Z

I pushed what I rewrote and just tested. It performed well for me and also implements the suggested changes. Feel free to take a look.

bmbouter added 3.0 labels Oct 31, 2018

bmbouter force-pushed the 4075 branch from 08ce1d5 to 2484e13 Compare October 31, 2018 21:59

dralley reviewed Oct 31, 2018

View reviewed changes

bmbouter force-pushed the 4075 branch from 2484e13 to c160942 Compare November 1, 2018 14:36

dkliban reviewed Nov 1, 2018

View reviewed changes

bmbouter force-pushed the 4075 branch from c160942 to 7dbc5e6 Compare November 1, 2018 16:20

dkliban approved these changes Nov 1, 2018

View reviewed changes

gmbnomis suggested changes Nov 1, 2018

View reviewed changes

bmbouter force-pushed the 4075 branch from 7dbc5e6 to adec0fb Compare November 1, 2018 19:50

dkliban approved these changes Nov 1, 2018

View reviewed changes

bmbouter merged commit a95ce0f into pulp:master Nov 3, 2018

bmbouter deleted the 4075 branch November 3, 2018 19:17

bmbouter added the Katello label Nov 5, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use force_close and concurrency restriction #3733

Use force_close and concurrency restriction #3733

bmbouter commented Oct 31, 2018 •

edited

Loading

pep8speaks commented Oct 31, 2018

dralley commented Oct 31, 2018

dralley Oct 31, 2018

bmbouter Oct 31, 2018

asmacdo Nov 1, 2018

bmbouter Nov 1, 2018

dralley commented Oct 31, 2018

dralley commented Oct 31, 2018

codecov bot commented Oct 31, 2018 •

edited

Loading

bmbouter commented Nov 1, 2018

bmbouter commented Nov 1, 2018

bmbouter commented Nov 1, 2018

dkliban Nov 1, 2018

bmbouter Nov 1, 2018

bmbouter commented Nov 1, 2018

bmbouter commented Nov 1, 2018

dkliban left a comment

gmbnomis left a comment

gmbnomis Nov 1, 2018 •

edited

Loading

bmbouter Nov 1, 2018

bmbouter Nov 1, 2018

bmbouter commented Nov 1, 2018

Use force_close and concurrency restriction #3733

Use force_close and concurrency restriction #3733

Conversation

bmbouter commented Oct 31, 2018 • edited Loading

pep8speaks commented Oct 31, 2018

dralley commented Oct 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dralley commented Oct 31, 2018

dralley commented Oct 31, 2018

codecov bot commented Oct 31, 2018 • edited Loading

Codecov Report

bmbouter commented Nov 1, 2018

bmbouter commented Nov 1, 2018

bmbouter commented Nov 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmbouter commented Nov 1, 2018

bmbouter commented Nov 1, 2018

dkliban left a comment

Choose a reason for hiding this comment

gmbnomis left a comment

Choose a reason for hiding this comment

gmbnomis Nov 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmbouter commented Nov 1, 2018

bmbouter commented Oct 31, 2018 •

edited

Loading

codecov bot commented Oct 31, 2018 •

edited

Loading

gmbnomis Nov 1, 2018 •

edited

Loading