Recover from disconnected SSH tunnel

[istreeter]

The loader opens a SSH tunnel immediately when it first starts up, and keeps the tunnel open indefinitely until the app terminates. But it's possible for the session to get disconnected, and the loader never discovers it has been disconnected.

This appears in the logs first of all like this:

INFO JCsh: Caught an exception, leaving main loop due to Connection reset
INFO JCsh: Disconnecting from port 22

And later lots and lots of messages like this:

ERROR Loader: Loading of <S3_URL> has failed. Adding intro retry queue. HikariPool-1 - Connection is not available, request timed out after 30001ms.

The underlying reason comes from these lines in the jcraft lib. Any exception on the socket is caught and handled and not raised to the application.

I suggest two fixes:

1. We should call session.setServerAliveInterval described here, which should make it much less likely that the tunnel gets disconnected.
2. When initiating the session, we should also start a fiber to periodically check session.isConnected.

I haven't worked out yet what action it should take if the session is disconnect: either restore the session, or raise an exception to crash the app. The former is more preferable if it works, but we should check that the JDBC connection recovers once the session is restored.