Handle issues with udsGracefulErrorHandling

[bdeitte]

See #123 (comment). It appears we need to handle more error cases for retries.

[msiebuhr]

Shouldn't we just try re-connecting on all of them.

And a important detail I noticed: Error-codes are negative. My code sees -102, -107 and -111.

[bdeitte]

@msiebuhr I see your comments in #114 (comment) as well so making this a more generic ticket on potential issues with this feature. Also interesting on what you see for codes, as it's different than what @alessio-modus mentions in #123

[pcothenet]

We're experiencing something like #114. When the agent restart, hot-shots never seem to reconnect properly (error -107)

[dhermes]

I'm currently looking into this as well. (I'll probably update this post as I go.) I've got a local-dd-agent Docker container / host UDS I've been using to reproduce this.

Error codes:

• 102 / 0x66 (Linux): ENETRESET
• 107 / 0x6b (Linux): ENOTCONN
• 111 / 0x6f (Linux): ECONNREFUSED (see also require('util').getSystemErrorName(-111))
• 54 / 0x36 (macOS): ECONNRESET, happens only on the first failure when UDS is deleted
• 39 / 0x27 (macOS): EDESTADDRREQ, happens for every attempt to send after deleting the initial UDS (even if a new file replaces it)

I see the handling of these send() errors just hardcodes 107 and 111, which is unfortunately platform specific.

[dhermes]

So here is a summary of the problems with the current implementation:

1. The error codes are negative (i.e. -107, -111) so no recovery will happen as written:

   hot-shots/lib/statsd.js

   Lines 128 to 129 in c488cc7

   	case 107:
   	case 111: {

2. If this.socket doesn't get created on start (as a UDS) then it will never recover (even if the UDS eventually gets fixed / recreated). This can be addressed by attempting to re-create a socket in sendMessage()

   hot-shots/lib/statsd.js

   Line 357 in c488cc7

   if (!this.socket) {

3. As mentioned above the list of errno codes are platform-specific to Linux

   hot-shots/lib/statsd.js

   Lines 128 to 129 in c488cc7

   	case 107:
   	case 111: {

   I recommend ditching the switch statement for a something like errorCodes.includes(-err.code) with errorCodes produced by

   function udsErrors() {
     if (process.platform == 'linux') {
       return [
         os.constants.errno.ENOTCONN,
         os.constants.errno.ECONNREFUSED,
       ]
     }
   
     if (process.platform == 'darwin') {
       return [
         os.constants.errno.EDESTADDRREQ,
         os.constants.errno.ECONNRESET,
       ]
     }
   
     // Unknown / not yet implemented
     return []
   }

4. When a new socket is created in the UDS recovery code

   hot-shots/lib/statsd.js

   Line 136 in c488cc7

   this.socket = createTransport(this, {

   it doesn't re-register socket error handling code, so this means a UDS failure can only be recovered from one time

I'll send a PR (based on dhermes#1) with fixes for all 4 but this may be a longer discussion