[securedrop-proxy] gracefully handle connection failures

[redshiftzero]

While using the Qubes proxy (via the SecureDrop SDK), I ran into the following exception:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/securedrop-proxy-3awDxAhP/lib/python3.5/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/home/user/.local/share/virtualenvs/securedrop-proxy-3awDxAhP/lib/python3.5/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/user/.local/share/virtualenvs/securedrop-proxy-3awDxAhP/lib/python3.5/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='REDACTED.onion', port=80): Max retries exceeded with url: /api/v1/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x737017375e10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))

We should handle this exception and return an informative response - I propose via a 500 status code and at least the explanation "connection error" in the request body. This can be used by clients to show informative feedback to the user, e.g. we can pop up something saying "Connection Error. Are you sure your workstation is connected to the internet?"

Once this is done, we can make the corresponding change in the SDK.

[redshiftzero]

We need to implement this: this is why in the client we see "Error in parsing JSON" in the logs when the server is down. We should provide a better response here as described above and then handle this case in the queue to ensure we pause processing until the server is back (i.e. if the server is down entirely, and not a timeout due to network delays).

This is also why in the test plan for freedomofpress/securedrop-client#578 (review) the queue doesn't appear to resume: this is because there isn't a timeout, there is an error in parsing this JSON response, and the queue isn't handling that case currently. You can reproduce it manually without the SDK in the client VM via:

echo '{"method": "GET", "path_query": "foo"}' | /usr/lib/qubes/qrexec-client-vm sd-proxy securedrop.Proxy

when the server is down: you'll see a crazy exception bubble up and be passed to the sd-svs VM