Update consul-k8s components to support Consul namespaces

This was a major change to the internals of most of the consul-k8s
commands. As part of the work, there were other changes that affect
folks not using namespaces as well. Details are broken down by process.

--> Catalog Sync

Namespaces:

This allows the catalog sync process to support Consul namespaces,
an Enterprise feature. It supports no namespaces (OSS), syncing k8s
services into a single Consul namespace and mirroring k8s namespaces
in Consul with an optional prefix.

Beyond namespaces:

It updates the settings for which k8s namespaces to sync. These
are now based on allow and deny lists, rather than the two previous
options of (1) a single k8s namespace, or (2) all k8s namespaces
except `kube-system`. This change is backwards compatible, however
if a user upgrades consul-k8s without upgrading the Helm chart as well,
there will be a slight difference in behavior for (2) in that it won't
automatically exclude `kube-system` on its own.

The underlying call to Consul to retrieve services has been switched
to retrieve services by the synthetic node `k8s-sync`. This causes a
slight behavior change in that we will no longer remove services with
the `ConsulK8STag` if it's not attached to the `k8s-sync` node.

Fixes a hot loop bug when getting an error from Consul when retrieving
service information.

Moves `c.sigCh` initialization to the init method to fix a race
condition occurring in tests.

Adds additional debug logging to resource.go and syncer.go.

--> ACL Bootstrapping

Namespaces:

Updates all policies that are created by the bootstrapper to include
namespace permissions as needed. Updates the Connect Injector's AuthMethod
to reflect the namespace registration settings (single destination,
mirroring, mirroring with prefix).

When namespaces are enabled, all policies and tokens for consul-k8s
components are being created within the `Consul` default namespace.
This is required for any cross-namespace permissions, and in the case
of catalog sync and the connect injector, the ability to create
Consul namespaces. Additionally, a specific cross-namespace policy
is created so that it can be attached to all created namespaces
to allow service discovery between Consul namespaces.

This makes sure all policies are updated if the acl bootstrapping
job is rerun, which happens on a helm upgrade. This allows someone
upgrading to a version that includes namespaces or changes their
namespacing config to also update the policies associated with
their acl tokens to reflect that change.

Beyond namespaces:

This separates auth method and binding rule checking logic.
If it exists already, binding rules are now always updated, which
supports config updates.

To make it easier to work with the code, it now uses a shared logger
and has been split into smaller files.

Updates mesh gateway acl policies with the correct permissions

-->  Connect Injector

Namespaces:

This adds namespace config options for registering injected
services into a single namespace as well as mirroring k8s
namespaces in Consul with an optional prefix.

It adds functionality to check for Consul namespace existence
and create new namespaces.

Service and proxy registration as well as service-defaults
have been updated to be namespace aware.

Adds additional parsing of the upstream annotation to support namespaces.
The format of the annotation becomes:

    `service_name.namespace:port:optional_datacenter`

The `service_name.namespace` is only parsed if namespaces are enabled. If
someone has added a `.namespace` in that case, the upstream will not work
correctly, as is the case where someone has put in an incorrect service
name, port or datacenter.

The upstream definitions in the service registration file includes the
namespace from the annotation. If it wasn't present in the annotation, no
namespace is included. This will automatically fallback to assuming the
service is in the same namespace as the service defining the upstream.

Beyond namespaces:

Updates the default envoy version to 1.13.0.

.circleci/config.yml

@@ -6,7 +6,8 @@ executors:
       - image: circleci/golang:1.13
     environment:
       - TEST_RESULTS: /tmp/test-results # path to where test results are saved
-      - CONSUL_VERSION: 1.6.3 # this can be OSS or enterprise, e.g., 1.7.0+ent-beta4
+      - CONSUL_VERSION: 1.7.0 # Consul's OSS version to use in tests
+      - CONSUL_ENT_VERSION: 1.7.0+ent # Consul's enterprise version to use in tests
 
 jobs:
   go-fmt-and-vet:
@@ -67,6 +68,34 @@ jobs:
       - store_artifacts:
           path: /tmp/test-results
 
+  test_enterprise:
+    executor: go
+    environment:
+      TEST_RESULTS: /tmp/test-results
+    parallelism: 1
+    steps:
+      - checkout
+      - run: mkdir -p $TEST_RESULTS
+
+      # Restore go module cache if there is one
+      - restore_cache:
+          keys:
+            - consul-k8s-modcache-v1-{{ checksum "go.mod" }}
+
+      # run go tests with gotestsum
+      - run: |
+          # download and install the consul binary
+          wget https://releases.hashicorp.com/consul/"${CONSUL_ENT_VERSION}"/consul_"${CONSUL_ENT_VERSION}"_linux_amd64.zip && \
+               unzip consul_"${CONSUL_ENT_VERSION}"_linux_amd64.zip -d /home/circleci/bin &&
+               rm consul_"${CONSUL_ENT_VERSION}"_linux_amd64.zip
+      - run: |
+          PACKAGE_NAMES=$(go list ./...)
+          gotestsum --junitfile $TEST_RESULTS/gotestsum-report.xml -- -tags=enterprise -p 4 $PACKAGE_NAMES
+      - store_test_results:
+          path: /tmp/test-results
+      - store_artifacts:
+          path: /tmp/test-results
+
   build-distros:
     executor: go
     environment:
@@ -94,6 +123,10 @@ workflows:
       - test:
           requires:
             - go-fmt-and-vet
+      - test_enterprise:
+          requires:
+            - go-fmt-and-vet
       - build-distros:
           requires:
             - test
+            - test_enterprise

Makefile

@@ -86,6 +86,10 @@ dev-tree:
 test:
 	go test ./...
 
+# requires a consul enterprise binary on the path
+ent-test:
+	go test ./... -tags=enterprise
+
 cov:
 	go test ./... -coverprofile=coverage.out
 	go tool cover -html=coverage.out

catalog/to-consul/consul_node_services_client.go

@@ -0,0 +1,114 @@
+package catalog
+
+import (
+	"fmt"
+
+	"github.com/hashicorp/consul/api"
+)
+
+// ConsulService is service registered in Consul.
+type ConsulService struct {
+	// Namespace is the Consul namespace the service is registered in.
+	// If namespaces are disabled this will always be the empty string even
+	// though the namespace is technically "default".
+	Namespace string
+	// Name is the name of the service in Consul.
+	Name string
+}
+
+// ConsulNodeServicesClient is used to query for node services.
+type ConsulNodeServicesClient interface {
+	// NodeServices returns consul services with the corresponding tag
+	// registered to the Consul node with nodeName. opts is used as the
+	// query options in the API call to consul. It returns the list of services
+	// (not service instances) and the query meta from the API call.
+	NodeServices(tag string, nodeName string, opts api.QueryOptions) ([]ConsulService, *api.QueryMeta, error)
+}
+
+// PreNamespacesNodeServicesClient implements ConsulNodeServicesClient
+// for Consul < 1.7 which does not support namespaces.
+type PreNamespacesNodeServicesClient struct {
+	Client *api.Client
+}
+
+// NodeServices returns Consul services tagged with
+// tag registered on nodeName using a Consul API that is supported in
+// Consul versions before 1.7. Consul versions after 1.7 still support
+// this API but the API is not namespace-aware.
+func (s *PreNamespacesNodeServicesClient) NodeServices(
+	tag string,
+	nodeName string,
+	opts api.QueryOptions) ([]ConsulService, *api.QueryMeta, error) {
+	// NOTE: We're not using tag filtering here so we can support Consul
+	// < 1.5.
+	node, meta, err := s.Client.Catalog().Node(nodeName, &opts)
+	if err != nil {
+		return nil, nil, err
+	}
+	if node == nil {
+		return nil, meta, nil
+	}
+
+	var svcs []ConsulService
+	// seenServices is used to ensure the svcs list is unique.
+	seenServices := make(map[string]bool)
+	for _, svcInstance := range node.Services {
+		svcName := svcInstance.Service
+		if _, ok := seenServices[svcName]; ok {
+			continue
+		}
+		for _, svcTag := range svcInstance.Tags {
+			if svcTag == tag {
+				if _, ok := seenServices[svcName]; !ok {
+					svcs = append(svcs, ConsulService{
+						// If namespaces are not enabled we use empty
+						// string.
+						Namespace: "",
+						Name:      svcName,
+					})
+					seenServices[svcName] = true
+				}
+				break
+			}
+		}
+	}
+	return svcs, meta, nil
+}
+
+// NamespacesNodeServicesClient implements ConsulNodeServicesClient
+// for Consul >= 1.7 which supports namespaces.
+type NamespacesNodeServicesClient struct {
+	Client *api.Client
+}
+
+// NodeServices returns Consul services tagged with
+// tag registered on nodeName using a Consul API that is supported in
+// Consul versions >= 1.7. If opts.Namespace is set to
+// "*", services from all namespaces will be returned.
+func (s *NamespacesNodeServicesClient) NodeServices(
+	tag string,
+	nodeName string,
+	opts api.QueryOptions) ([]ConsulService, *api.QueryMeta, error) {
+	opts.Filter = fmt.Sprintf("\"%s\" in Tags", tag)
+	nodeCatalog, meta, err := s.Client.Catalog().NodeServiceList(nodeName, &opts)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	var svcs []ConsulService
+	// seenServices is used to ensure the svcs list is unique. Its keys are
+	// <namespace>/<service name>.
+	seenSvcs := make(map[string]bool)
+	for _, svcInstance := range nodeCatalog.Services {
+		svcName := svcInstance.Service
+		key := fmt.Sprintf("%s/%s", svcInstance.Namespace, svcName)
+		if _, ok := seenSvcs[key]; !ok {
+			svcs = append(svcs, ConsulService{
+				Namespace: svcInstance.Namespace,
+				Name:      svcName,
+			})
+			seenSvcs[key] = true
+		}
+	}
+	return svcs, meta, nil
+}