Revert "Revert "allow self signed certs with insecureSkipVerify""

[s12chung]

Reverts #1776, which is a revert of #1769

I'd like to bring this code up, as it'd be great if we could have this feature. I don't think I can put in the time to support support custom CA bundles per #1027 (comment) now. I understand that insecureSkipVerify is insecure--hence the warnings. If I can get the code in, awesome. Otherwise, our team will live with our fork until we move away from self-signed certs.

Can we discuss about this topic on this PR?

relates to #1027

[skriss]

@s12chung based on all of the discussions, we've decided that we'll go ahead with getting this PR merged. I'll take a look at the code later today and provide any feedback. We may want to tweak the messages/documentation to make it super-clear that this is an insecure option.

cc @nrb @carlisia @prydonius

[prydonius]

We talked about potentially removing the insecure option from the server, and only allowing insecure connections from the client. This way, you would have to use a self-signed cert for the server -> object store connection, but in the client you can use --insecure-skip-tls-verify for debugging purposes. Would that work for you @s12chung?

[s12chung]

@prydonius I believe the server would need access to a secret that contains the cert. I'm good with that :) I just don't have the time to implement/test it on our end right now.

[prydonius]

@s12chung yeah, we would need a separate PR to accept a self-signed cert and add that to the certificate chain. In this PR, we should just remove the insecure option for the server for now. Good to hear that would work for you, I think that will mitigate the attack surface of this change.

[s12chung]

@prydonius without the insecure option on the server, I think the server would run into the same error the client does. unless

I believe the server would need access to a secret that contains the cert.

[skriss]

The plan for this is:

• once comments are addressed and PR is rebased, merge as-is
• when Support custom CA bundles for object storage connections #1027 is addressed, consider removing one or both of the insecure options (client to server, server to object store)

@carlisia, since this is also touching the AWS cloud provider code, need your input on whether we can merge this into master now-ish, or whether we need to split out the AWS-specific pieces into a separate PR for the new repo.

[s12chung]

rebased. changed the changelog and added the suggested changes.

[skriss]

LGTM.

[skriss]

@nrb @carlisia @prydonius PTAL.

[carlisia]

Lgtm. 👍