Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement heapq for cookie expire times #9203

Merged
merged 19 commits into from
Sep 21, 2024
Merged

Implement heapq for cookie expire times #9203

merged 19 commits into from
Sep 21, 2024

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Sep 19, 2024

What do these changes do?

Implement heapq for cookie expire times

Are there changes in behavior for the user?

no

Is it a substantial burden for the maintainers to support this?

heapq is a bit of a complex structure to maintain

Related issue number

fixes #8575
fixes #7790

benchmark script (we care about the filter_other_domain case for expires and that still includes the filtering time so its actually even faster): #7790 (comment)

Before
filter_other_domain: 0.451234458014369

After
filter_other_domain: 0.0018771658651530743

note that profile is not exactly the same number of iterations since its using a real world use case of 60s and its hard to get it exactly the same (1644 before) / (1413 after) but its such a significant difference, its more than enough to show the performance improvement of ~96-97% less run time (adjusted ratio for the iterations):

before
do_expiration_before

after
do_expiration_after

aiohttp/cookiejar.py Outdated Show resolved Hide resolved
@bdraco bdraco added backport-3.10 Trigger automatic backporting to the 3.10 release branch by Patchback robot backport-3.11 Trigger automatic backporting to the 3.11 release branch by Patchback robot labels Sep 19, 2024
Copy link

codecov bot commented Sep 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.32%. Comparing base (8911419) to head (57344db).
Report is 1 commits behind head on master.

✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff            @@
##           master    #9203    +/-   ##
========================================
  Coverage   98.31%   98.32%            
========================================
  Files         107      107            
  Lines       34510    34635   +125     
  Branches     4100     4113    +13     
========================================
+ Hits        33929    34055   +126     
  Misses        410      410            
+ Partials      171      170     -1     
Flag Coverage Δ
CI-GHA 98.22% <100.00%> (+<0.01%) ⬆️
OS-Linux 97.88% <100.00%> (+0.01%) ⬆️
OS-Windows 96.31% <100.00%> (+0.01%) ⬆️
OS-macOS 97.56% <100.00%> (+0.01%) ⬆️
Py-3.10.11 97.66% <100.00%> (+0.01%) ⬆️
Py-3.10.15 97.60% <100.00%> (+0.01%) ⬆️
Py-3.11.10 97.49% <100.00%> (+0.01%) ⬆️
Py-3.11.9 97.55% <100.00%> (+0.01%) ⬆️
Py-3.12.6 97.94% <100.00%> (+0.01%) ⬆️
Py-3.9.13 97.55% <100.00%> (+0.01%) ⬆️
Py-3.9.20 97.49% <100.00%> (+0.01%) ⬆️
Py-pypy7.3.16 97.10% <100.00%> (+0.01%) ⬆️
VM-macos 97.56% <100.00%> (+0.01%) ⬆️
VM-ubuntu 97.88% <100.00%> (+0.01%) ⬆️
VM-windows 96.31% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bdraco added a commit that referenced this pull request Sep 19, 2024
noticed while working on #9203

TODO: show profiles
bdraco added a commit that referenced this pull request Sep 19, 2024
noticed while working on #9203

TODO: show profiles
aiohttp/cookiejar.py Outdated Show resolved Hide resolved
@psf-chronographer psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Sep 20, 2024
@bdraco bdraco marked this pull request as ready for review September 20, 2024 16:30
@bdraco
Copy link
Member Author

bdraco commented Sep 21, 2024

Running well on production overnight 👍

@bdraco bdraco merged commit 6e70c0a into master Sep 21, 2024
34 of 35 checks passed
@bdraco bdraco deleted the cookie_heap branch September 21, 2024 09:05
Copy link
Contributor

patchback bot commented Sep 21, 2024

Backport to 3.10: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 6e70c0a on top of patchback/backports/3.10/6e70c0ab91f383a0d28448d1506d826a9b44e53d/pr-9203

Backporting merged PR #9203 into master

  1. Ensure you have a local repo clone of your fork. Unless you cloned it
    from the upstream, this would be your origin remote.
  2. Make sure you have an upstream repo added as a remote too. In these
    instructions you'll refer to it by the name upstream. If you don't
    have it, here's how you can add it:
    $ git remote add upstream https://github.com/aio-libs/aiohttp.git
  3. Ensure you have the latest copy of upstream and prepare a branch
    that will hold the backported code:
    $ git fetch upstream
    $ git checkout -b patchback/backports/3.10/6e70c0ab91f383a0d28448d1506d826a9b44e53d/pr-9203 upstream/3.10
  4. Now, cherry-pick PR Implement heapq for cookie expire times #9203 contents into that branch:
    $ git cherry-pick -x 6e70c0ab91f383a0d28448d1506d826a9b44e53d
    If it'll yell at you with something like fatal: Commit 6e70c0ab91f383a0d28448d1506d826a9b44e53d is a merge but no -m option was given., add -m 1 as follows instead:
    $ git cherry-pick -m1 -x 6e70c0ab91f383a0d28448d1506d826a9b44e53d
  5. At this point, you'll probably encounter some merge conflicts. You must
    resolve them in to preserve the patch from PR Implement heapq for cookie expire times #9203 as close to the
    original as possible.
  6. Push this branch to your fork on GitHub:
    $ git push origin patchback/backports/3.10/6e70c0ab91f383a0d28448d1506d826a9b44e53d/pr-9203
  7. Create a PR, ensure that the CI is green. If it's not — update it so that
    the tests and any other checks pass. This is it!
    Now relax and wait for the maintainers to process your pull request
    when they have some cycles to do reviews. Don't worry — they'll tell you if
    any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

Copy link
Contributor

patchback bot commented Sep 21, 2024

Backport to 3.11: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 6e70c0a on top of patchback/backports/3.11/6e70c0ab91f383a0d28448d1506d826a9b44e53d/pr-9203

Backporting merged PR #9203 into master

  1. Ensure you have a local repo clone of your fork. Unless you cloned it
    from the upstream, this would be your origin remote.
  2. Make sure you have an upstream repo added as a remote too. In these
    instructions you'll refer to it by the name upstream. If you don't
    have it, here's how you can add it:
    $ git remote add upstream https://github.com/aio-libs/aiohttp.git
  3. Ensure you have the latest copy of upstream and prepare a branch
    that will hold the backported code:
    $ git fetch upstream
    $ git checkout -b patchback/backports/3.11/6e70c0ab91f383a0d28448d1506d826a9b44e53d/pr-9203 upstream/3.11
  4. Now, cherry-pick PR Implement heapq for cookie expire times #9203 contents into that branch:
    $ git cherry-pick -x 6e70c0ab91f383a0d28448d1506d826a9b44e53d
    If it'll yell at you with something like fatal: Commit 6e70c0ab91f383a0d28448d1506d826a9b44e53d is a merge but no -m option was given., add -m 1 as follows instead:
    $ git cherry-pick -m1 -x 6e70c0ab91f383a0d28448d1506d826a9b44e53d
  5. At this point, you'll probably encounter some merge conflicts. You must
    resolve them in to preserve the patch from PR Implement heapq for cookie expire times #9203 as close to the
    original as possible.
  6. Push this branch to your fork on GitHub:
    $ git push origin patchback/backports/3.11/6e70c0ab91f383a0d28448d1506d826a9b44e53d/pr-9203
  7. Create a PR, ensure that the CI is green. If it's not — update it so that
    the tests and any other checks pass. This is it!
    Now relax and wait for the maintainers to process your pull request
    when they have some cycles to do reviews. Don't worry — they'll tell you if
    any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

bdraco added a commit that referenced this pull request Sep 21, 2024
Co-authored-by: Sam Bull <[email protected]>
(cherry picked from commit 6e70c0a)
bdraco added a commit that referenced this pull request Sep 21, 2024
Co-authored-by: Sam Bull <[email protected]>
(cherry picked from commit 6e70c0a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-3.10 Trigger automatic backporting to the 3.10 release branch by Patchback robot backport-3.11 Trigger automatic backporting to the 3.11 release branch by Patchback robot bot:chronographer:provided There is a change note present in this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Expiring the cookie jar is unexpectedly expensive Remove _next_expiration field from CookieJar
2 participants