[BUG] Saltstack uses ancient Tornado with inefficient buffer management

[szjur]

Description
Is there any specific reason Saltstack clings onto ancient Tornado with inefficient buffer management? In 2017 Tornado got improvements which eliminated costly memory copy operations:

tornadoweb/tornado#2147
tornadoweb/tornado#2169

While analysing memory leaks in Salt master with tracemalloc salt/ext/tornado/iostream.py was identified as the main memory hog. Indeed planting some debugs there I could see that _write_buffer could easily reach hundreds of megabytes under high load of large events. The legacy tornado logic for shrinking the buffer is not too aggressive, which directly contributes to high memory consumption:

https://github.com/saltstack/salt/blob/master/salt/ext/tornado/iostream.py#L872

if self._write_buffer_pos > self._write_buffer_size:

If my logic is correct, this will only shrink the buffer if it's at least half empty.

Setup
Irrelevant - applies to latest code from Saltstack repo.

Steps to Reproduce the behavior
Impose very high load of big events on Salt and see poor iostream performance.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report
Any

[OrangeDog]

The developers have been trying for years to upgrade Tornado (e.g. #56169), and have never managed to finish the work.

The current plan appears to be to drop Tornado entirely: #61380

[szjur]

Asyncio seems reasonable, fingers crossed :-) That Tornado improvement involving _StreamBuffer class was only introduced in 5.0. I probably won't dare backporting it but I'll probably have a go at tweaking the condition to release the unused parts of _write_buffer sooner.