Static listener with tls_context support?

[phedoreanu]

Hey guys,

When I enable tls_context for a static listener rotor fails to unmarshal the file

could not deserialize static resources: json: cannot unmarshal string into Go value of type []json.RawMessage

• static config

listeners:
- address:
    socket_address:
      address: 0.0.0.0
      port_value: 443
  filter_chains:
  - filters:
    - name: envoy.http_connection_manager
      config:
        codec_type: AUTO
        stat_prefix: ingress_http
        route_config:
          virtual_hosts:
          - name: backend
            domains:
            - "example.com"
            routes:
            - match:
                prefix: "/service/1"
              route:
                cluster: service1
            - match:
                prefix: "/service/2"
              route:
                cluster: service2
        http_filters:
        - name: envoy.router
          config: {}
    tls_context:
      common_tls_context:
        alpn_protocols: h2,http/1.1
        tls_params:
          tls_minimum_protocol_version: TLSv1_2
          tls_maximum_protocol_version: TLSv1_3
          cipher_suites: ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
        tls_certificates:
        - certificate_chain: { filename: /etc/envoy/cert.crt }
          private_key: { filename: /etc/envoy/cert.key }

Any ideas?
Thanks

[phedoreanu]

Found it!

    tls_context:
      common_tls_context:
        alpn_protocols: ["h2,http/1.1"]
        tls_params:
          tls_minimum_protocol_version: TLSv1_2
          tls_maximum_protocol_version: TLSv1_3
          cipher_suites: ["ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"]
        tls_certificates:
        - certificate_chain: { filename: /etc/envoy/cert.crt }
          private_key: { filename: /etc/envoy/cert.key }```

[phedoreanu]

@9len Why does only the default-cluster get a DynamicRouteConfiguration?

How can I add one/generate one for a static listener?

[9len]

Can you be a little more concrete about what you're trying to do, and what you've tried?

are you trying to add a static listener that points to RDS? If so, you should be able to copy the RDW config out of the default-cluster listener; you can run rotor-test-clientin the container to dump the contents.

[phedoreanu]

I'm trying to route traffic from the default-cluster, which I can't modify, to a new static listener.

I've tried everything. So far the best 💡 I think is to optimise the mergeBehavior.

Right now, I'm testing with ROTOR_XDS_STATIC_RESOURCES_CONFLICT_BEHAVIOR=overwrite

Adding:

        rds:
          config_source:
            api_config_source:
              api_type: GRPC
              cluster_names: ["tbn-xds"]
              refresh_delay: 30.000s
          route_config_name: default-cluster:443

works! 🎉

[9len]

Just so I understand, you want make it so that the listener does TLS, but still uses the rotor-generated routes to the rotor-collected clusters?

I was able to get the following to work:

---
listeners:
- name: default-cluster:80
  address:
    socketAddress:
      address: 0.0.0.0
      portValue: 80
  filterChains:
  - filterChainMatch:
    filters:
    - name: envoy.http_connection_manager
      config:
        http_filters:
        - config: {}
          name: envoy.cors
        - config: {}
          name: envoy.router
        rds:
          config_source:
            api_config_source:
              api_type: GRPC
              grpc_services:
              - envoy_grpc:
                  cluster_name: tbn-xds
              refresh_delay: 30.000s
          route_config_name: default-cluster:80
        stat_prefix: default-cluster-80
    tls_context:
      common_tls_context:
        alpn_protocols: ["h2,http/1.1"]
        tls_params:
          tls_minimum_protocol_version: TLSv1_2
          tls_maximum_protocol_version: TLSv1_3
          cipher_suites: ["[ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305]", "[ECDHE-RSA-AES128-GCM-SHA256|ECDHE-RSA-CHACHA20-POLY1305]"]
        tls_certificates:
        - certificate_chain: { filename: /etc/envoy/cert.crt }
          private_key: { filename: /etc/envoy/cert.key }

with -xds.static-resources.conflict-behavior overwrite

The Rotor RDS only serves routes for default-cluster:80, it doesn't know anything about static listeners or clusters. So I think your best bet is to do the above, then run your envoy container so that it maps port 443 to port 80. Alternatively, you can make a new listener, but you can only specify routes inline in that case.

Also note that your cipher_suites value was malformed, each suite needs to be in its own string in the array.

[phedoreanu]

Yes, one listener with TLS, but still uses the rotor-generated routes to the rotor-collected clusters and the static routes defined.

I've got it now somewhat to work, but when it starts up it randomly picks between rds and route_config. Can it do both?

With 2 listeners on different ports, both routes (rds and route_config) work fine.

[9len]

you can only specify fixed routes in a listener or RDS, not both.

[phedoreanu]

Ok, no problem. I'll add the new tag on the old Services to migrate them.

Thanks!

[phedoreanu]

The Rotor RDS only serves routes for default-cluster:80, it doesn't know anything about static listeners or clusters.

From the rotor README:

Routes (RDS) are created from your clusters. Each cluster is exposed via a single domain with the same names as the cluster, and a single catch-all route (/).

default-cluster:443 is fine, but can I make rotor change the domain? Maybe by adding a new url tag?
I can't use the domain name as the Service/cluster name, because it's not allowed by Nomad 😞

[
  {
    "ID": "9d1f1d33-3543-88d8-8eae-24770d46be7a",
    "Node": "ip-172-32-18-98.eu-central-1.compute.internal",
    "Address": "172.32.18.98",
    "Datacenter": "dc1",
    "TaggedAddresses": {
      "lan": "172.32.18.98",
      "wan": "172.32.18.98"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceID": "_nomad-task-jcskbneazntvlwe4st4da3ru5onpmtnr",
    "ServiceName": "test-group-task",
    "ServiceTags": [
      "xds-cluster",
      "url=test-group-task.example.com"
    ],
    "ServiceAddress": "172.32.18.98",
    "ServicePort": 25631,
    "ServiceEnableTagOverride": false,
    "CreateIndex": 22139688,
    "ModifyIndex": 22139688
  }
]

So I think your best bet is to do the above, then run your envoy container so that it maps port 443 to port 80.

Got around that with ROTOR_XDS_STANDALONE_PORT: '443'

Thanks for the cipher_suites btw!

EDIT: added Consul response.

[phedoreanu]

Got it working with #25 😃