Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test and eliminate additional sources of cyclic garbage #2063

Merged
merged 4 commits into from
Jul 18, 2021

Conversation

richardsheridan
Copy link
Contributor

@richardsheridan richardsheridan commented Jul 14, 2021

from MultiErrorCatcher.__exit__, _multierror.copy_tb, MultiError.filter, CancelScope.__exit__, and NurseryManager.__aexit__ methods. This was nearly impossible to catch until #1864 landed so it wasn't cleaned up in #1805 (or during the live stream: https://www.youtube.com/watch?v=E_jvJVYXUAk).

@belm0: curious if this shows up in your application benchmarking.

@codecov
Copy link

codecov bot commented Jul 14, 2021

Codecov Report

Merging #2063 (e48d638) into master (2b33fe6) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #2063   +/-   ##
=======================================
  Coverage   99.56%   99.56%           
=======================================
  Files         114      114           
  Lines       14634    14688   +54     
  Branches     1119     1120    +1     
=======================================
+ Hits        14570    14624   +54     
  Misses         43       43           
  Partials       21       21           
Impacted Files Coverage Δ
trio/_core/_multierror.py 98.91% <100.00%> (+0.01%) ⬆️
trio/_core/_run.py 100.00% <100.00%> (ø)
trio/_core/tests/test_multierror.py 99.30% <100.00%> (+0.03%) ⬆️
trio/_core/tests/test_run.py 100.00% <100.00%> (ø)

trio/_core/tests/test_run.py Outdated Show resolved Hide resolved
@belm0
Copy link
Member

belm0 commented Jul 14, 2021

@belm0: curious if this shows up in your application benchmarking.

I haven't been keeping an eye on this lately, but I don't think it came up. Generally we try to avoid internal async API's that raise exceptions because multi-errors are so hard to deal with.

@richardsheridan
Copy link
Contributor Author

Just noticed that CancelScope.__exit__ was mentioned as another possible source of cycles but it doesn't show up in the test yet. Happy to hear suggestions, or else I'll poke at it this weekend.

@richardsheridan
Copy link
Contributor Author

I found a gross refcycle in the KI system. Here is the output of objgraph.show_backrefs:

graph

Note that the args tuple is referenced both by the frame locals as well as the f_locals dict, leading to the spooky code at

trio/trio/_core/_ki.py

Lines 162 to 168 in f9d0e2a

# don't create a refcycle if an exception is passed through and
# re-raised, see test_cancel_scope_exit_doesnt_create_cyclic_garbage
del args, kwargs
# deep magic to remove refs via f_locals
locals()
# TODO: check if PEP558 changes the need for this call
# https://github.com/python/cpython/pull/3640

an alternative solution might be to inline the locals()["@TRIO_KI_PROTECTION_ENABLED"]=True in CancelScope.__exit__, as that is a special case; that way we don't have to run extra bytecode every time KI protection is toggled.

maybe some change would make the checkpoint insufficient. weakref will notice that the nursery is still alive!
Copy link
Member

@belm0 belm0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an alternative solution might be to inline the locals()["@TRIO_KI_PROTECTION_ENABLED"]=True in CancelScope.exit, as that is a special case; that way we don't have to run extra bytecode every time KI protection is toggled.

this is appealing-- perhaps add another commit implementing it and let njs weigh in

trio/_core/_run.py Outdated Show resolved Hide resolved
@belm0
Copy link
Member

belm0 commented Jul 18, 2021

LGTM. Add a newsfragment and check that PR description matches latest changes?

@richardsheridan richardsheridan changed the title test and eliminate an additional source of cyclic garbage test and eliminate additional sources of cyclic garbage Jul 18, 2021
@belm0 belm0 merged commit 3f4e1a2 into python-trio:master Jul 18, 2021
@richardsheridan richardsheridan deleted the cyclic_garbage_continued branch July 19, 2021 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants