Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unclosed socket from luigi.rpc module #3173

Closed
starhel opened this issue Jun 9, 2022 · 0 comments · Fixed by #3175
Closed

Unclosed socket from luigi.rpc module #3173

starhel opened this issue Jun 9, 2022 · 0 comments · Fixed by #3175

Comments

@starhel
Copy link
Contributor

starhel commented Jun 9, 2022

Description
Luigi does not close all connections to remote scheduler:

ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=(...), raddr=(...)>

This is probably connected to not closed requests.Session objects in RequestsFetcher.

luigi/luigi/rpc.py

Lines 100 to 118 in 78145d6

class RequestsFetcher:
def __init__(self, session):
from requests import exceptions as requests_exceptions
self.raises = requests_exceptions.RequestException
self.session = session
self.process_id = os.getpid()
def check_pid(self):
# if the process id change changed from when the session was created
# a new session needs to be setup since requests isn't multiprocessing safe.
if os.getpid() != self.process_id:
self.session = requests.Session()
self.process_id = os.getpid()
def fetch(self, full_url, body, timeout):
self.check_pid()
resp = self.session.post(full_url, data=body, timeout=timeout)
resp.raise_for_status()
return resp.text

Packages

urllib3==1.26.9
requests==2.27.1
luigi==3.0.3 (same code on master)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant