Orderly router shutdown and resource cleanup

[kgiusti]

On shutdown the router stops all processes "mid stream" and attempts to manually clean up resources that remain in use. This is a best effort attempt and involves bespoke cleanup code for just about every system in the router. Here is an example where the router core attempts to release resources for all links that were active at the point of shutdown.

This is error prone and triggers many leak events in ASAN. This makes it nearly impossible to determine if a leak occurs during runtime (probably a serious issue) or just at shutdown (sloppy, but not service affecting).

The router's shutdown needs to be refactored to instead perform an orderly shutdown controlled by management. This feature would cause management to:

• close the $management subscription to prevent further external management access.
• delete all listeners and connectors to prevent new connections being established.
• close all active connections
• stop any periodic services (idle link scrubber, stuck delivery detection, etc).

This will trigger the existing run-time cleanup for all outstanding messages, dispositions, etc. Once all connections finish closing, all in-flight messages are released, and other provisioned entities and services have been released, the subsystems (core, proactor, etc) can clean up and shutdown.

The devil is in the details for sure - this will be no small job.

[kgiusti]

Needs fixing: #156

[kgiusti]

Goal: We want avoid runtime leaks in the router

Old way: All leak analysis at process exit (by asan or alloc pool leak detection)

New way: Shutdown all listeners, terminate connections, and OTHER STUFF; and then,
run leak analysis at a time prior to process exit.
The key thing here is to detect leaks only for the desired scope, not for the whole router process.

[jiridanek]

I'm sorry but I still don't much understand this.

• Runtime leaks are quite different problem from orderly shutdown. You can have one and not the other, vice versa. Consider the memory leaks you can have in Java and JavaScript, for example. (You keep reference to something you should not, or somebody else keeps reference on your behalf (how insidious!), and so gc cannot gc it. Or you keep reference longer than necessary, so things get eventually collected, but you might get OOM in the meantime.)
• I do not see what the new way would translate into, practically. How does it apply to, for example, Mick's asan leak in qd_policy_c_counts_alloc #253 or to Leaked termini and qdr_link_t at router shutdown #335?

[jiridanek]

Even simpler trial case of a leak to be judged by the new rules, #156

[jiridanek]

The key thing here is to detect leaks only for the desired scope, not for the whole router process.

I guess this is the main stumbling block for me. How do you do that? How do you tell that "this buffer was 'leaked' because we did not bother to free it (yet). How can you know it is not truly leaked because there is a bug in the code and it would never be cleared up, even if the connection it is related to was closed (while the router continues running)"?

[kgiusti]

Won't fix.

It has been decided to maintain the current resource cleanup architecture