Check agent behavior with goleak

[tonyredondo]

We have to check the agent goroutines lifecycle with goleak, and document the results

[drodriguezhdez]

Identified some goroutines:

go.undefinedlabs.com/scopeagent/agent.(*SpanRecorder).loop

go.undefinedlabs.com/scopeagent.Run.func1

internal/poll.runtime_pollWait

net/http.(*http2ClientConn).roundTrip

Apart from identifying the goroutines and document it, probably we'd need to think if it is possible to avoid those goroutines, cause some customers are worried about missing legitimate goroutines leaks if they ignore all these.

[tonyredondo]

Mmm, probably the first one could be avoided with the wrapper.

The second one its the one created to wait for the SIGTERM signal in order to clear the buffer. Maybe could be avoided too using the wrapper, because we can remove the buffer.

I wonder if the third one is related to the first one I don’t know.

[drodriguezhdez]

The 3rd one I think is related to the http2Transport.newClientConn

[Goroutine 13 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
goroutine 13 [IO wait]:
internal/poll.runtime_pollWait(0x1f79cc8, 0x72, 0xffffffffffffffff)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0001e2398, 0x72, 0x1700, 0x174f, 0xffffffffffffffff)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001e2380, 0xc000220000, 0x174f, 0x174f, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_unix.go:169 +0x201
net.(*netFD).Read(0xc0001e2380, 0xc000220000, 0x174f, 0x174f, 0x203000, 0x125f800, 0xc00020a138)
	/usr/local/Cellar/go/1.14/libexec/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000010018, 0xc000220000, 0x174f, 0x174f, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc00029e0a0, 0xc000220000, 0x174f, 0x174f, 0x19, 0x1590, 0xc0001079c8)
	/usr/local/Cellar/go/1.14/libexec/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc00020a258, 0x15b1900, 0xc00029e0a0, 0x100bf25, 0x149a540, 0x14f75e0)
	/usr/local/Cellar/go/1.14/libexec/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc00020a000, 0x9100028, 0xc000010018, 0x5, 0xc000010018, 0x9)
	/usr/local/Cellar/go/1.14/libexec/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00020a000, 0x0, 0x0, 0xc000107d38)
	/usr/local/Cellar/go/1.14/libexec/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/usr/local/Cellar/go/1.14/libexec/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc00020a000, 0xc000142000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc0007018c0, 0xc00019e2d8, 0x9, 0x9, 0xc000107d38, 0x152eb00, 0x12eb195)
	/usr/local/Cellar/go/1.14/libexec/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x15b17a0, 0xc0007018c0, 0xc00019e2d8, 0x9, 0x9, 0x9, 0xc0000a0050, 0x0, 0x15b1a00)
	/usr/local/Cellar/go/1.14/libexec/src/io/io.go:310 +0x87
io.ReadFull(...)
	/usr/local/Cellar/go/1.14/libexec/src/io/io.go:329
net/http.http2readFrameHeader(0xc00019e2d8, 0x9, 0x9, 0x15b17a0, 0xc0007018c0, 0x0, 0x0, 0xc00029a1b0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/h2_bundle.go:1479 +0x87
net/http.(*http2Framer).ReadFrame(0xc00019e2a0, 0xc00029a1b0, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/h2_bundle.go:1737 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc000107fa8, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/h2_bundle.go:8246 +0x8d
net/http.(*http2ClientConn).readLoop(0xc000682300)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/h2_bundle.go:8174 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/Cellar/go/1.14/libexec/src/net/http/h2_bundle.go:7174 +0x64a
]