better handling of unrecoverable network errors

[bmwill]

Context from a slack thread:

From @bmwill

In working through this I've identified a potential issue with how we are handling some errors cases when sending messages to other peers, in particular if we try to send a message to the other side, and the message sending was successful but there was an RPC error (eg you sent the message to the wrong route) then we ignore this even though we know then that the message send will never be "successful".

From @huitseeker

Indeed.
The current signature of send_message is

send_message(
        &mut self,
        address: Multiaddr,
        message: BincodeEncodedPayload,
    ) -> CancelOnDropHandler<Result<tonic::Response<types::Empty>, anyhow::Error>>

Where Result<tonic::Responsetypes::Empty, anyhow::Error> is aliased as MessageResult.
If you know how to refactor the Error case of this MessageResult type into a Sentinel error that outputs essentially one variant chosen between RecoverableError, IrrecoverableError, the downstream changes to retry behavior are easy and I'm happy to do them.