Using a HTTP proxy sometimes breaks TLS connections

[erikjohnston]

Stack trace, with (I think) related log lines, though there are multiple HTTP connections happening at once:

2021-02-16 08:51:51,072 [14936] DEBUG sygnal.helper.proxy.proxy_asyncio Initiating proxy CONNECT
2021-02-16 08:51:51,092 [14936] DEBUG sygnal.helper.proxy.proxy_asyncio CONNECT response from proxy: hv=b'HTTP/1.0', r=b'200', rp=b'Connection established'
2021-02-16 08:51:51,092 [14936] DEBUG sygnal.helper.proxy.proxy_asyncio Ready to switch over protocol
2021-02-16 08:51:51,679 [14936] DEBUG aioapns Remote setting changed: ChangedSetting(setting=SettingCodes.HEADER_TABLE_SIZE, original_value=4096, new_value=4096)
2021-02-16 08:51:51,680 [14936] DEBUG aioapns Remote setting changed: ChangedSetting(setting=SettingCodes.MAX_CONCURRENT_STREAMS, original_value=None, new_value=1)
2021-02-16 08:51:51,680 [14936] DEBUG aioapns Remote setting changed: ChangedSetting(setting=SettingCodes.MAX_FRAME_SIZE, original_value=16384, new_value=16384)
2021-02-16 08:51:51,680 [14936] DEBUG aioapns Remote setting changed: ChangedSetting(setting=SettingCodes.MAX_HEADER_LIST_SIZE, original_value=None, new_value=8000)
2021-02-16 08:51:51,680 [14936] ERROR asyncio application protocol failed to receive SSL data
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f54156d1d50>
transport: <_SelectorSocketTransport fd=23 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/sslproto.py", line 545, in data_received
    self._app_protocol.data_received(chunk)
  File "/opt/matrix/sygnal/lib/python3.7/site-packages/aioapns/connection.py", line 100, in data_received
    self.flush()
  File "/opt/matrix/sygnal/lib/python3.7/site-packages/aioapns/connection.py", line 199, in flush
    self.transport.write(self.conn.data_to_send())
AttributeError: 'NoneType' object has no attribute 'write'

This appears to be due to using a HTTP proxy, my hunch is that there is a race where the protocol gets data during start_tls:

sygnal/sygnal/helper/proxy/proxy_asyncio.py

Lines 152 to 161 in ab87ab8

	transport = await self._event_loop.start_tls(
	self._transport,
	new_protocol,
	self._sslcontext,
	server_hostname=self._target_hostport[0],
	)
	# start_tls does NOT call connection_made on new_protocol, so we
	# must do it ourselves
	new_protocol.connection_made(transport)

We have some tests for making plain TCP connections with a proxy, so I think the first step here is to add a test with TLS connections and see if that breaks (there is some prior art in the unit tests for making TLS connections).