Add support for reading Auth data from a file on disk

[schancel]

This commit adds support from reading the authorization section of the
gnatsd config from a JSON file located within the nats-operator container.
This allows secrets the data to be placed on disk using sidecars like
consul-template.

Depends on #171

[schancel]

Untested, and needs a test. What do you recommend here @wallyqs ?

[schancel]

Implements #156 ?

[schancel]

Okay, if this looks good, I'll try to write some tests. Looks pretty gnarly though.

[schancel]

Also, should I put the sample NatsCluster file somewhere?

[wallyqs]

Thanks @schancel ! fwiw I think this approach might better than the other one of injecting all the credentials into the same config secret. The test could be a variation of this one maybe: https://github.com/nats-io/nats-operator/blob/master/test/e2e/config_reload_test.go#L100
It would be great to have an example here to include in the release notes too: https://github.com/nats-io/nats-operator/tree/master/example

[schancel]

@wallyqs Well, I finally got it compiling. I still can't setup the e2e tests locally due to virtualization being disabled on the hardware I'm using.

Do you have any idea what's going on here w/ the panic and failing test?

[schancel]

Okay it's failing on:

	// Wait for the single pod to be created.
	ctx1, fn := context.WithTimeout(context.Background(), waitTimeout)
	defer fn()
	if err = f.WaitUntilFullMeshWithVersion(ctx1, natsCluster, size, version); err != nil {
		t.Fatal(err)
	}

Specifically for my test. There must be something wrong with the manifest being provided to kubernetes.

[wallyqs]

I built a new version of the reloader and the following works for me:

# This is an example NatsCluster manifest which uses a 3rd party initContainer
# to fetch the authorization credentials from outside kubernetes.
#
# An example of this could be consul-template getting user/passwords from vault.
apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
  name: nats-auth-file-example
  namespace: default
spec:
  size: 1
  version: "1.4.1"

  natsConfig:
    maxPayload: 20971520

  pod:
    enableConfigReload: true
    reloaderImage: "wallyqs/nats-server-config-reloader"
    reloaderImageTag: "0.4.4-v1alpha2"
    reloaderImagePullPolicy: "IfNotPresent"

    volumeMounts:
      - name: authconfig
        mountPath: /etc/nats-config/authconfig

  auth:
    clientsAuthFile: "authconfig/auth.json"

  template:
    spec:
      initContainers:
        - name: secret-getter
          image: "busybox"
          command: ["sh", "-c", "echo 'users = [ { user: 'foo', pass: 'bar' } ]' > /etc/nats-config/authconfig/auth.json"]
          volumeMounts:
            - name: authconfig
              mountPath: /etc/nats-config/authconfig
      volumes:
        - name: authconfig
          emptyDir: {}

[wallyqs]

The reloader image does not get build along with the test so currently have to specify the new reloader image that can be used, for example using these that I pushed:

    reloaderImage: "wallyqs/nats-server-config-reloader"
    reloaderImageTag: "0.4.5-v1alpha2"

[wallyqs]

For the include to work it has to be mounted at /etc/nats-config/authconfig/ since the includes have to be relative to where the main config is located which is /etc/nats-config/.

[schancel]

😱

That's a huge caveat to add to the documentation. Will definitely cause confusion.

[wallyqs]

Agree... we'd have to cover with documentation for now.

[wallyqs]

I might have been wrong on this one... it looks like the reloader would already follow the events in the folder so if the volume is mounted underneath the main nats config file, it would signal as well in case of changes:

https://github.com/nats-io/nats-operator/blob/master/pkg/reloader/reloader.go#L94-L99

[schancel]

Also, it's not supposed to be recursively watching. (According to the documentation)

I created another PR to fix these problems with reloader independently.

[wallyqs]

That's right... it only works per folder, it is not recursive so would need to list each of the folders that should be tracked for changes.

[schancel]

@wallyqs Yeah, I noticed that. I'm updating the reloader now to be a bit more generic

[wallyqs]

Once the multi-reloader branch is merged and this branch is rebased, and updating the reloader command, applying the following to the branch makes things work for me: wallyqs@35ed435

There was an issue in the reloader test that when decoding JSON it would also accidentally escape > because by default the Go encoder escapes HTML, only workaround seems to be is to setup manually the encoder to avoid escaping html...

@@ -211,11 +215,24 @@ func ConfigReloadTestHelper(t *testing.T, customizer NatsClusterCustomizerWSecre
 			},
 		},
 	}
-	// Serialize the object containing authentication data.
-	if d, err = json.Marshal(auth); err != nil {
+	// Serialize the object containing authentication data,
+	// we are using wildcard so need to unescape the HTML
+	// which the JSON encoder does by default...
+	buf := &bytes.Buffer{}
+	encoder := json.NewEncoder(buf)
+	encoder.SetEscapeHTML(false)
+	err = encoder.Encode(auth)
+	if err != nil {
+		t.Fatal(err)
+	}
+	buf2 := &bytes.Buffer{}
+	err = json.Indent(buf2, buf.Bytes(), "", "  ")
+	if err != nil {
 		t.Fatal(err)
 	}
+
 	// Create a secret containing authentication data.
+	d = buf2.Bytes()
 	if cas, err = f.CreateSecret(f.Namespace, "data", d); err != nil {
 		t.Fatal(err)
 	}

[wallyqs]

Merged via 7ad614f