Fixes some bugs in the limit connection feature #2964

pfreixes · 2018-04-28T06:50:36Z

There are two places which both try to release the next waiter. When the ongoing connection is being canceled calling explicitly to wake up a waiter and when the connection is being released, either because was called explicitly or implicitly due to the close of a response.

In both situations, we call the same method _release_waiter that will iterate per all of the
queues till found one that has a waiter that is not already canceled.

This iteration will also check that the limit and the limit per host is not overcome. The logic to check how many available connections/free slots there are has been placed behind the new _available_connections function.

So far this PR fixes:

Take into account the limit and limit per host at any time that we try
to wake up a waiter. No connection available, no wake-up.
Don't give up till reach a waiter that is not finished.
Iterate always through all of the different hosts.

There are two places which try to release the next waiter, when the ongoing connection is being canceled and we call explicitly to wake up a waiter or when the connection is being released. In both situations we call the same method `_release_waiter` that will iterate per all of the queues till found one that has a waiter that is not already canceled. This iteration will also check that the limit and the limit per host is not overcomed. The logic to check how many available connections/free slots there are has been placed behind the `_available_connections` function. So far this PR fixes: - Take into account the limit and limit per host at any time that we try to wake up a waiter. No availables, no wake up. - Dont give up till reach a waiter that is not finished. - Iterate always through all of the different hosts. What is missing: - More coverage - Shuffle the keys to give always the same chances to all of the hosts?

pfreixes · 2018-04-28T06:54:09Z

Having the concern of #1821, no degradation perceived with this PR

imit_semaphore:
  Responses: Counter({200: 9999})
  Took:      20.797451972961426 seconds
limit_builtin:
  Responses: Counter({200: 9999})
  Took:      20.224246501922607 seconds

pfreixes · 2018-04-28T06:54:47Z

Also worth it if @fafhrd91 can take a look.

codecov-io · 2018-04-28T07:36:59Z

Codecov Report

Merging #2964 into master will increase coverage by 0.17%.
The diff coverage is 96.66%.

@@            Coverage Diff             @@
##           master    #2964      +/-   ##
==========================================
+ Coverage   97.98%   98.16%   +0.17%     
==========================================
  Files          40       41       +1     
  Lines        7592     7842     +250     
  Branches     1323     1392      +69     
==========================================
+ Hits         7439     7698     +259     
+ Misses         51       47       -4     
+ Partials      102       97       -5

Impacted Files	Coverage Δ
aiohttp/connector.py	`98.16% <96.66%> (+1.29%)`	⬆️
aiohttp/__init__.py	`100% <0%> (ø)`
aiohttp/web_app.py	`99.16% <0%> (+0.06%)`	⬆️
aiohttp/web_urldispatcher.py	`99.6% <0%> (+0.5%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ba7bf1...ff39bb8. Read the comment docs.

asvetlov · 2018-04-29T05:28:09Z

I'm at a conference, sorry.
Will try to find a time for review tomorrow.

thehesiod · 2018-05-01T08:36:14Z

will try to carve out some time too later today, was working on a bunch of aiobotocore PRs that just landed, am hoping to switch to that and this fix in prod for the perf improvement.

thehesiod · 2018-05-01T08:38:48Z

btw I was thinking, should a "FIFO semaphore" abstraction be refactored from this? I think it would make the main code clearer.

asvetlov · 2018-05-02T08:50:59Z

@thehesiod What "FIFO semaphore" abstraction is?

asvetlov · 2018-05-02T09:00:57Z

aiohttp/connector.py

                # remove a waiter even if it was cancelled, normally it's
                #  removed when it's notified
                try:
                    waiters.remove(fut)
                except ValueError:  # fut may no longer be in list
                    pass

-                if not waiters:
+                if key in self._waiters and not self._waiters[key]:


What is the case when waiters is not self._waiters[key]?

nothing, fixed!

asvetlov · 2018-05-02T09:01:35Z

aiohttp/connector.py

+        # Having the dict keys ordered this avoids to iterate
+        # at the same order at each call.
+        queues = list(self._waiters.keys())
+        random.shuffle(queues)


Why do we need shuffling?
The operation is not free.

Yes, that hurts me. But without the shuffle the iteration of the queues will be done always in the same order, having chances of starvation for certain queues.

We could try to get rid of that operation, but it will imply some important changes.

The idea of @thehesiod of having a "true" FIFO would get rid of this operation, but my gut feeling says that this FIFO queue will have enough footprint - has to take care of the limit per host - to make it slower than just a call shuffle.

PD: a shuffle operation takes around 1 microsecond for a list of 10 elements.

waiters is a deque.
deque.rotate() is cheap. Can we use it?

Remember that _waiters is the dictionary that has the queues. So, unfortunately we can't use it.

We might implement the FIFO on top a unique deque using the rotate method as a way to skip a waiter that can't be deallocated because of the limit per host restriction. I didn't know but the slice of the head element is O(1), so the check and later the skip can be done without losing the deque properties.

Got it. The PR's code is ok to me.

asvetlov · 2018-05-02T09:02:40Z

aiohttp/connector.py

+                waiter = waiters.popleft()
+                if not waiter.done():
+                    waiter.set_result(None)
+                    found = True


Isn't break enough? It saves a check.

The break does not work. Anyway changed to avoid the found check duplicating some code.

thehesiod · 2018-05-02T19:35:12Z

@asvetlov basically what this code is trying to do is be a semaphore on the connectors, with the caveat that the waiters are released in order. So why I said a FIFO Semaphore :)

asvetlov · 2018-05-02T19:50:15Z

@thehesiod got you

asvetlov · 2018-05-02T21:59:06Z

aiohttp/connector.py

+
+                    return
+
+            if not waiters:


The condition is always true.

I mean it would be nice to have a test case where waiters become non-empty after releasing.

Yes it doesn't make sense right now let me get rid of this part of the code and I will check the tests.

pfreixes · 2018-05-03T06:17:33Z

aiohttp/connector.py

                # remove a waiter even if it was cancelled, normally it's
                #  removed when it's notified
                try:
                    waiters.remove(fut)
                except ValueError:  # fut may no longer be in list
                    pass

+                raise e
+            finally:


@asvetlov can you check this new change? My concern with the previous implementation was because having a del self._waiters[key] in two places was a tick bomb and hard to maintain.

pfreixes · 2018-05-04T05:50:08Z

@asvetlov any clue about how can I increase the coverage?

asvetlov · 2018-05-04T06:58:22Z

I think how everything is perfect.

asvetlov · 2018-05-04T07:00:08Z

thank you, @pfreixes

pfreixes · 2018-05-04T07:01:45Z

Thanks to you, any chance to get a new bugfix release?

thehesiod · 2018-05-04T07:03:22Z

sorry I didn't have a chance to help review, been in a killer time sink project

asvetlov · 2018-05-04T07:08:02Z

@thehesiod no problem.
If you see something -- we can fix it in a separate PR.

lock · 2019-10-28T05:04:49Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a [new issue] for related bugs.
If you feel like there's important points made in this discussion, please include those exceprts into that [new issue].
[new issue]: https://github.com/aio-libs/aiohttp/issues/new

pfreixes requested review from asvetlov, hellysmile and thehesiod April 28, 2018 06:50

Added changes file

58495e3

pfreixes added 2 commits April 29, 2018 14:22

Refine some tests

96acca9

Shuffle queues to avoid starvation

199b2bd

asvetlov reviewed May 2, 2018

View reviewed changes

Fixes some useless code

b94c684

asvetlov approved these changes May 2, 2018

View reviewed changes

Increse coverage

ad92b5e

asvetlov reviewed May 2, 2018

View reviewed changes

Evict and del the key waiters only in one place

9e54abf

pfreixes commented May 3, 2018

View reviewed changes

pfreixes added 5 commits May 3, 2018 15:54

Fixed an scenario that might end up with a KeyError sitaution

38011f4

Increase coverage

ab2a5f6

Fix test

69bf66a

Add more coverage for connection

9e958ff

Fix test

ff39bb8

asvetlov merged commit 9116128 into master May 4, 2018

asvetlov deleted the fix_buggy_connection_limit branch May 4, 2018 06:59

asvetlov mentioned this pull request May 4, 2018

Degrading performance over time... #1821

Closed

lock bot added the outdated label Oct 28, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 28, 2019

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Oct 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes some bugs in the limit connection feature #2964

Fixes some bugs in the limit connection feature #2964

pfreixes commented Apr 28, 2018 •

edited

Loading

pfreixes commented Apr 28, 2018

pfreixes commented Apr 28, 2018

codecov-io commented Apr 28, 2018 •

edited

Loading

asvetlov commented Apr 29, 2018

thehesiod commented May 1, 2018

thehesiod commented May 1, 2018

asvetlov commented May 2, 2018

asvetlov May 2, 2018

pfreixes May 2, 2018

asvetlov May 2, 2018

pfreixes May 2, 2018

pfreixes May 2, 2018 •

edited

Loading

asvetlov May 2, 2018

pfreixes May 2, 2018

asvetlov May 2, 2018

asvetlov May 2, 2018

pfreixes May 2, 2018

thehesiod commented May 2, 2018

asvetlov commented May 2, 2018

asvetlov May 2, 2018

asvetlov May 2, 2018

pfreixes May 3, 2018

pfreixes May 3, 2018

asvetlov May 3, 2018

pfreixes commented May 4, 2018

asvetlov commented May 4, 2018

asvetlov commented May 4, 2018

pfreixes commented May 4, 2018

thehesiod commented May 4, 2018

asvetlov commented May 4, 2018

lock bot commented Oct 28, 2019

Fixes some bugs in the limit connection feature #2964

Fixes some bugs in the limit connection feature #2964

Conversation

pfreixes commented Apr 28, 2018 • edited Loading

pfreixes commented Apr 28, 2018

pfreixes commented Apr 28, 2018

codecov-io commented Apr 28, 2018 • edited Loading

Codecov Report

asvetlov commented Apr 29, 2018

thehesiod commented May 1, 2018

thehesiod commented May 1, 2018

asvetlov commented May 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pfreixes May 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thehesiod commented May 2, 2018

asvetlov commented May 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pfreixes commented May 4, 2018

asvetlov commented May 4, 2018

asvetlov commented May 4, 2018

pfreixes commented May 4, 2018

thehesiod commented May 4, 2018

asvetlov commented May 4, 2018

lock bot commented Oct 28, 2019

pfreixes commented Apr 28, 2018 •

edited

Loading

codecov-io commented Apr 28, 2018 •

edited

Loading

pfreixes May 2, 2018 •

edited

Loading