Change reporting to TLS

[rainest]

What this PR does / why we need it:
Makes reporting connections over TLS.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[rainest]

I am doing something that breaks the test TLS connection. Possibly wrapping its accept in a goroutine is freezing it somehow? Not sure why. It receives a TLS client hello, but never responds with its own server hello. The struct indicates that it has not completed any handshakes:

timeout.pcap.gz
conn.txt

[rainest]

Deadlock resolved by manually handshaking. I'm not particularly fond of the connection re-establishment behavior, but not immediately seeing how we can handle it better since we don't get an affirmative "yes, the connection is closed" net.ErrClosed, and what we do get isn't exported and would require us to scrape the syscall text. Is there some better way to determine that upstream closed the connection?

> github.com/kong/kubernetes-ingress-controller/v2/internal/util.Reporter.send() ./reports.go:105 (hits goroutine(12):3 total:3) (PC: 0x1a449c8)
	err: error(*crypto/tls.permanentError) *{
		err: net.Error(*net.OpError) *{
			Op: "write",
			Net: "tcp",
			Source: net.Addr(*net.TCPAddr) ...,
			Addr: net.Addr(*net.TCPAddr) ...,
			Err: error(*os.SyscallError) ...,},}
   100:	
   101:	func (r Reporter) send(signal string, uptime int) {
   102:		message := "<14>signal=" + signal + ";uptime=" +
   103:			strconv.Itoa(uptime) + ";" + r.serializedInfo
   104:		_, err := r.conn.Write([]byte(message))
=> 105:		if err != nil {
   106:			r.Logger.Errorf("failed to send report: %s", err)
   107:			r.conn = nil
   108:		}
   109:	}
(dlv) call err.Error()
> github.com/kong/kubernetes-ingress-controller/v2/internal/util.Reporter.send() ./reports.go:105 (PC: 0x1a449c8)
Values returned:
	~r0: "write tcp 127.0.0.1:46284->127.0.0.1:61833: write: broken pipe"

   100:	
   101:	func (r Reporter) send(signal string, uptime int) {
   102:		message := "<14>signal=" + signal + ";uptime=" +
   103:			strconv.Itoa(uptime) + ";" + r.serializedInfo
   104:		_, err := r.conn.Write([]byte(message))
=> 105:		if err != nil {
   106:			r.Logger.Errorf("failed to send report: %s", err)
   107:			r.conn = nil
   108:		}
   109:	}

[github-actions]

Licenses differ between commit 17d8820 and base:

+++ pr_licenses.csv	2021-12-14 00:50:51.734731162 +0000
@@ -8,12 +8,7 @@
 github.com/bombsimon/logrusr,https://github.com/bombsimon/logrusr/blob/master/LICENCE,MIT
 github.com/cenkalti/backoff/v4,https://github.com/cenkalti/backoff/blob/master/v4/LICENSE,MIT
 github.com/cespare/xxhash/v2,https://github.com/cespare/xxhash/blob/master/v2/LICENSE.txt,MIT
-github.com/containerd/containerd,https://github.com/containerd/containerd/blob/master/LICENSE,Apache-2.0
 github.com/davecgh/go-spew/spew,https://github.com/davecgh/go-spew/blob/master/spew/LICENSE,ISC
-github.com/docker/distribution,https://github.com/docker/distribution/blob/master/LICENSE,Apache-2.0
-github.com/docker/docker,https://github.com/docker/docker/blob/master/LICENSE,Apache-2.0
-github.com/docker/go-connections,https://github.com/docker/go-connections/blob/master/LICENSE,Apache-2.0
-github.com/docker/go-units,https://github.com/docker/go-units/blob/master/LICENSE,Apache-2.0
 github.com/evanphx/json-patch,https://github.com/evanphx/json-patch/blob/master/LICENSE,BSD-3-Clause
 github.com/evanphx/json-patch/v5,https://github.com/evanphx/json-patch/blob/master/v5/LICENSE,BSD-3-Clause
 github.com/fatih/color,https://github.com/fatih/color/blob/master/LICENSE.md,MIT
@@ -44,8 +39,6 @@
 github.com/mitchellh/reflectwalk,https://github.com/mitchellh/reflectwalk/blob/master/LICENSE,MIT
 github.com/modern-go/concurrent,https://github.com/modern-go/concurrent/blob/master/LICENSE,Apache-2.0
 github.com/modern-go/reflect2,https://github.com/modern-go/reflect2/blob/master/LICENSE,Apache-2.0
-github.com/opencontainers/go-digest,https://github.com/opencontainers/go-digest/blob/master/LICENSE,Apache-2.0
-github.com/opencontainers/image-spec/specs-go,https://github.com/opencontainers/image-spec/blob/master/specs-go/LICENSE,Apache-2.0
 github.com/pkg/errors,https://github.com/pkg/errors/blob/master/LICENSE,BSD-2-Clause
 github.com/prometheus/client_golang/prometheus,https://github.com/prometheus/client_golang/blob/master/prometheus/LICENSE,Apache-2.0
 github.com/prometheus/client_model/go,https://github.com/prometheus/client_model/blob/master/go/LICENSE,Apache-2.0```

[rainest]

Realized we don't actually need persistent connections and interruption handling: while the test interval is 1 second, and the actual interval does include time.Second, the actual interval is 3600 * time.Second, not just time.Second.

Dunno if Splunk will even let us keep the idle connection for an hour, but there's no real need to, since reopening it once each hour doesn't raise the same performance issues as doing it once each second, and drastically simplifies testing.

This is ready for review, but we want to conduct a practical test where we run an image built from this branch against the actual reporting server to confirm we receive stats via the new listen. I need to get permissions to access that.

[shaneutt]

Looks good, only things before merging:

• if there's no reason NOT to set MinVersion: tls.VersionTLS13, then let's just set that
• let's make sure we take care of deadlines and write verifications for the tls connection.

[rainest]

#2104 seems quite determined to fail here. Wait for that fix, merge main, and this should pass.

[shaneutt]

actually the most recent failure seemed unrelated:

        	            	Internal error occurred: failed calling webhook "validations.kong.konghq.com": Post "https://webhook-kong-validations-gateway.kong-system.svc:443/?timeout=10s": dial tcp 10.96.85.183:443: connect: connection refused

[rainest]

The heck, auto-merge merged it despite the failure?

Webhook thing looks like we probably need a connection test eventually on the webhook service before trying actually create resources.