DB containers produce ResourceWarning: unclosed <socket...>

[sha1n]

Describe the bug

DbContainer instances may trigger a warning regarding unclosed socket. This is caused by the fact that DbContainer._connect creates an sqlalchemy engine and leaves it hanging after its done. I managed to reproduce it many time on an M1 macbook pro and fix it by changing the code to dispose the engine right before _connect returns.

Here is what the warning looks like:

../../../Library/Caches/pypoetry/virtualenvs/python-testkit-5lwKVr0p-py3.10/lib/python3.10/site-packages/_pytest/runner.py:341:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../Library/Caches/pypoetry/virtualenvs/python-testkit-5lwKVr0p-py3.10/lib/python3.10/site-packages/_pytest/runner.py:262: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
../../../Library/Caches/pypoetry/virtualenvs/python-testkit-5lwKVr0p-py3.10/lib/python3.10/site-packages/pluggy/_hooks.py:433: in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
../../../Library/Caches/pypoetry/virtualenvs/python-testkit-5lwKVr0p-py3.10/lib/python3.10/site-packages/pluggy/_manager.py:112: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
../../../Library/Caches/pypoetry/virtualenvs/python-testkit-5lwKVr0p-py3.10/lib/python3.10/site-packages/_pytest/unraisableexception.py:88: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
E
E               Traceback (most recent call last):
E                 File "/Users/sha1n/Library/Caches/pypoetry/virtualenvs/python-testkit-5lwKVr0p-py3.10/lib/python3.10/site-packages/requests/adapters.py", line 149, in __init__
E                   super().__init__()
E               ResourceWarning: unclosed <socket.socket fd=19, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/sha1n/.docker/run/docker.sock>

../../../Library/Caches/pypoetry/virtualenvs/python-testkit-5lwKVr0p-py3.10/lib/python3.10/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

To Reproduce

I run pytest with the following ini options and it fails every few runs

filterwarnings = [
    "error", # treating all warnings as errors by default
]

Runtime environment

$ uname -a
Darwin Shais-MBP.lan 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

Get the python version.

$ python --version

Python 3.10.12

$ docker info

Client:
 Version:    24.0.5
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     /Users/sha1n/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     /Users/sha1n/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/sha1n/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/sha1n/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/sha1n/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/sha1n/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/sha1n/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     /Users/sha1n/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 15
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 9.718GiB
 Name: docker-desktop
 ID: 463da969-2d68-4402-8b75-681e617ebbfb
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

$ poetry show

poetry show
astroid                 2.15.6    An abstract syntax tree for Python with inference support.
black                   23.7.0    The uncompromising code formatter.
certifi                 2023.7.22 Python package for providing Mozilla's CA Bundle.
charset-normalizer      3.2.0     The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
click                   8.1.6     Composable command line interface toolkit
coverage                7.3.0     Code coverage measurement for Python
deprecation             2.1.0     A library to handle automated deprecations
dill                    0.3.7     serialize all of Python
docker                  6.1.3     A Python library for the Docker Engine API.
exceptiongroup          1.1.3     Backport of PEP 654 (exception groups)
faker                   19.3.0    Faker is a Python package that generates fake data for you.
idna                    3.4       Internationalized Domain Names in Applications (IDNA)
iniconfig               2.0.0     brain-dead simple config-ini parsing
isort                   5.12.0    A Python utility / library to sort Python imports.
lazy-object-proxy       1.9.0     A fast and thorough lazy object proxy.
mccabe                  0.7.0     McCabe checker, plugin for flake8
mypy                    1.5.0     Optional static typing for Python
mypy-extensions         1.0.0     Type system extensions for programs checked with the mypy type checker.
packaging               23.1      Core utilities for Python packages
pathspec                0.11.2    Utility library for gitignore style pattern matching of file paths.
platformdirs            3.10.0    A small Python package for determining appropriate platform-specific dirs, e.g. a "user data dir".
pluggy                  1.2.0     plugin and hook calling mechanisms for python
psycopg2-binary         2.9.7     psycopg2 - Python-PostgreSQL Database Adapter
pylint                  2.17.5    python code static checker
pytest                  7.4.0     pytest: simple powerful testing with Python
pytest-asyncio          0.21.1    Pytest support for asyncio
pytest-cov              4.1.0     Pytest plugin for measuring coverage.
python-dateutil         2.8.2     Extensions to the standard Python datetime module
requests                2.31.0    Python HTTP for Humans.
six                     1.16.0    Python 2 and 3 compatibility utilities
sqlalchemy              2.0.19    Database Abstraction Library
testcontainers          3.7.1     Library provides lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container
testcontainers-core     0.0.1rc1  Core component of testcontainers-python.
testcontainers-postgres 0.0.1rc1  PostgreSQL component of testcontainers-python.
tomli                   2.0.1     A lil' TOML parser
tomlkit                 0.12.1    Style preserving TOML library
typing-extensions       4.7.1     Backported and Experimental Type Hints for Python 3.7+
urllib3                 2.0.4     HTTP library with thread-safe connection pooling, file post, and more.
websocket-client        1.6.1     WebSocket client for Python with low level API options
wrapt                   1.15.0    Module for decorators, wrappers and monkey patching.

[rstcruzo]

I'm seeing unclosed socket warnings too, but seems like mine were caused by connections to docker. Here's the PR to fix it: #380

[totallyzen]

Hey @rstcruzo and @sha1n thanks for the reports and PRs!
I've noted your PRs, will schedule to integrate them after #408 has been carried out to ease some fixes