aiohttp client session. it doesn't close a connection after keepalive_timeout

[jaemyunlee]

Long story short

I created test server with Asyncio Protocol and Transport and created client side with aiohttp client. If I make multiple requests, It doesn't close connection. (If I make just a few requests, It closes connection, but If i make requests more, It doesn't close and leave it connected)

Expected behaviour

I expected that it closes a connection after keepalive_timeout(default 15 or custom value)

Actual behaviour

It doesn't close and leave it connected

Steps to reproduce

1. run server.py
2. change forloop in client.py : range(1,3)
3. check connection_lost callback invoke : it works fine
4. change forloop in client.py : range(1,10)
5. check connection_lost callback invoke : it doesn't invoke connection_lost.

# server.py

import asyncio
import logging
import sys
import time

SERVER_ADDRESS = ('localhost', 8888)

logging.basicConfig(
    level=logging.DEBUG,
    format='%(name)s: %(message)s',
    stream=sys.stderr
)
log = logging.getLogger('main')

event_loop = asyncio.get_event_loop()

class CustomServer(asyncio.Protocol):

    def connection_made(self, transport):

        self.transport = transport
        self.address = transport.get_extra_info('peername')
        self.log = logging.getLogger('custom')
        self.log.debug('connection accepted')
        self.log.debug(self.address)

    def data_received(self, data):
        self.log.debug(f'receive {data}')
        body = 'hello world!'.encode()
        time.sleep(1)
        self.transport.write((
            b'HTTP/1.1 200 OK\r\n'
            b'Content-Length: %b\r\n'
            b'Content-Type: text/plain\r\n\r\n'
            b'%b') % (str(len(body)).encode(), body)
        )

    def connection_lost(self, exc):
        if exc:
            self.log.error(f'Error: {exc}')
        else:
            self.log.debug('closing')
        super().connection_lost(exc)


factory = event_loop.create_server(CustomServer, *SERVER_ADDRESS)
server = event_loop.run_until_complete(factory)
log.debug('server started')

try:
    event_loop.run_forever()
finally:
    log.debug('closing server')
    server.close()
    event_loop.run_until_complete(server.wait_closed())
    log.debug('closing event loop')
    event_loop.close()

# client.py

import socket

import asyncio
import aiohttp

event_loop = asyncio.get_event_loop()
timeout = aiohttp.ClientTimeout(total=60)
session = aiohttp.ClientSession(connector=aiohttp.TCPConnector(keepalive_timeout=5, family=socket.AF_INET), timeout=timeout)

async def request_example():
    resp = await session.get('http://localhost:8888/')
    res = await resp.text()
    return res

async def main():
    try:
        for idx in range(1, 7):
            res = await request_example()
            print(f'{idx} : {res}')
    except asyncio.TimeoutError:
        print('timeout')


event_loop.run_until_complete(main())
print([task for task in asyncio.Task.all_tasks() if not task.done()])
print(session.closed)
event_loop.run_forever()

Your environment

case 1
aiohttp version : 3.4.4 Client
OS : Mac Pro(High Sierra)
python : 3.6

case2
aiohttp version : 3.4.4 Client
OS : Ubuntu14.04
python : 3.5

• changed f string in the code to format to run with python 3.5

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #2849 (Getting Connection closed with aiohttp client), #3175 (aiohttp client timeout does not work), #3052 (SSL with closed connections), #2827 (Exception handling doesn't close connection), and #965 (ClientSession.close() does not await for self._connector.close()).

[Sadbot]

The same error in pytest tests

[bitphage]

Regression has been introduces in 964921d in BaseConnector._cleanup:

@@ -197,43 +206,29 @@ class BaseConnector(object):
         """Cleanup unused transports."""
         if self._cleanup_handle:
             self._cleanup_handle.cancel()
-            self._cleanup_handle = None

With such a change, in BaseConnector._release() condition if self._cleanup_handle is None: is only True when first invoked. In subsequent calls self._cleanup_handle is something like <TimerHandle cancelled when=5853434>, so cleanup task is not being started.