bundled ca is getting replaced with CA created for RabbitMQ in vault

[sandhraprakash]

Hi,
I am using Hashicorp vault to generate certs for RabbitMQ. I followed tutorial as per https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/vault-tls.

After deploying rabbitmq in etc/rabbitmq-tls I can see ca.crt, tls.crt and tls.key. tls.crt and tls.key are as expected. But I was expecting to see the ca bundle I uploaded in kubernetes secret in ca.crt. Instead I got the CA that signed tls.crt and tls.key over there.
Without bundled CA I am not able to connect to rabbitmq.

Below is the rabbitmq.yaml

kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  image: rabbitmq:3.9.13
  secretBackend:
    vault:
      role: rabbitmq
      tls:
        ipSans: 127.0.0.1
        altNames: localhost
        pkiIssuerPath: pkirabbitmqserver/issue/cert-issuer        
  rabbitmq:
    additionalPlugins:
      - rabbitmq_mqtt
      - rabbitmq_management
      - rabbitmq_auth_mechanism_ssl
      - rabbitmq_auth_backend_oauth2
    additionalConfig: |
      ssl_options.fail_if_no_peer_cert = true
      ssl_options.verify = verify_peer
      log.connection.level = debug
      mqtt.listeners.ssl.default = 8883
      mqtt.listeners.tcp.default = 1883
      log.console.level = debug
      auth_backends.1 = rabbit_auth_backend_oauth2
      auth_backends.2 = rabbit_auth_backend_internal
      ssl_cert_login_from = common_name
      mqtt.ssl_cert_login = true
      auth_oauth2.https.peer_verification = verify_peer
      auth_oauth2.https.depth = 1
      auth_oauth2.https.fail_if_no_peer_cert = true
      auth_oauth2.https.hostname_verification = wildcard
    advancedConfig: |
      [
        {rabbitmq_auth_backend_oauth2, [
          {resource_server_id, <<"rabbitmq">>},
          {key_config, [
          {jwks_url, <<"http://localhost:8585/auth/realms/RabbitMQ/protocol/openid-connect/certs">>}
          ]}
        ]} 
      ].    
  tls:
    disableNonTLSListeners: true
    #secretName: tls-secret
    caSecretName: ca-secret

Kubectl describe of the server gives insights into the annotations in which ca.crt has issuing ca in it.

$ kubectl describe pod/rabbitmq-server-0  -n rabbitmq-system
Name:         rabbitmq-server-0
Namespace:    rabbitmq-system
Priority:     0
Node:         minikube/192.168.49.2
Start Time:   Wed, 01 Jun 2022 15:41:25 +0530
Labels:       app.kubernetes.io/component=rabbitmq
              app.kubernetes.io/name=rabbitmq
              app.kubernetes.io/part-of=rabbitmq
              controller-revision-hash=rabbitmq-server-754cc995fd
              statefulset.kubernetes.io/pod-name=rabbitmq-server-0
Annotations:  prometheus.io/port: 15691
              prometheus.io/scrape: true
              rabbitmq.com/lastRestartAt: 2022-06-01T10:11:12Z
              vault.hashicorp.com/agent-init-first: true
              vault.hashicorp.com/agent-inject: true
              vault.hashicorp.com/agent-inject-secret-ca.crt: pkirabbitmqserver/issue/cert-issuer
              vault.hashicorp.com/agent-inject-secret-tls.crt: pkirabbitmqserver/issue/cert-issuer
              vault.hashicorp.com/agent-inject-secret-tls.key: pkirabbitmqserver/issue/cert-issuer
              vault.hashicorp.com/agent-inject-status: injected
              vault.hashicorp.com/agent-inject-template-ca.crt:

                {{- with secret "pkirabbitmqserver/issue/cert-issuer" "common_name=rabbitmq.rabbitmq-system.svc" "alt_names=rabbitmq-server-0.rabbitmq-nod...
                {{ .Data.issuing_ca }}
                {{- end }}
              vault.hashicorp.com/agent-inject-template-tls.crt:

                {{- with secret "pkirabbitmqserver/issue/cert-issuer" "common_name=rabbitmq.rabbitmq-system.svc" "alt_names=rabbitmq-server-0.rabbitmq-nod...
                {{ .Data.certificate }}
                {{- end }}
              vault.hashicorp.com/agent-inject-template-tls.key:

                {{- with secret "pkirabbitmqserver/issue/cert-issuer" "common_name=rabbitmq.rabbitmq-system.svc" "alt_names=rabbitmq-server-0.rabbitmq-nod...
                {{ .Data.private_key }}
                {{- end }}
              vault.hashicorp.com/role: rabbitmq
              vault.hashicorp.com/secret-volume-path-ca.crt: /etc/rabbitmq-tls
              vault.hashicorp.com/secret-volume-path-tls.crt: /etc/rabbitmq-tls
              vault.hashicorp.com/secret-volume-path-tls.key: /etc/rabbitmq-tls

How can I have bundled ca in ca.crt instead? Any insight is appreciated

[MarcialRosales]

Hi @sandhraprakash , what you are experiencing is what it is expected. When RabbitMQ requests a private-pubic-key pair, it gets a private key, a certificate file and a CA certificate file that issued/signed the certificate.

Can you elaborate how you are trying to reach RabbitMQ and what failures/errors you are getting?

If you were authenticating via oauth2 protocol, I noticed in your advanced.config that you have configured jwks_url to go to localhost. Is this expected?

[sandhraprakash]

Hello @MarcialRosales,

We are using 2 types of authentication, rabbit_auth_backend_oauth2 (I have configured keycloak locally at localhost:8585) and rabbit_auth_backend_internal. I am trying to connect an MQTTX client with RabbitMQ using tls certificate authentication, the certificate common_name is added as User in RabbitMQ. In this case it will get authenticated by rabbit_auth_backend_internal.

Prior to vault integration I was able to connect the client to RabbitMQ. The ca.crt used to reflect the content of the ca-secret (bundled ca). Now its getting replaced by the issuing ca.
Which results in the below error from MQTTX Client

[2022-06-09 16:58:51] [INFO] Connect client local, MQTT/SSL connection: mqtts://localhost:8883
[2022-06-09 16:58:51] [ERROR] local connect fail, MQTT.js onError trigger, Error: self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)
    at TLSSocket.emit (events.js:315:20)
    at TLSSocket._finishInit (_tls_wrap.js:932:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)
[2022-06-09 16:58:51] [INFO] local connect close, MQTT.js onClose trigger

If this is the expected result after vault integration how RabbitMQ will handle certificate based authentication? how and where to provide bundled ca?

[MarcialRosales]

What it is expected is that RabbitMQ takes the ca certificate from Vault along with the private key and the server's certificate. RabbitMQ operator does not take ca certificates from anywhere else when Vault is enabled.

I believe the issue you are having could be that your client application's certificates were issued by another CA certificate which is not the CA certificate Vault is using. Are you using vault api to request a private-public-key pair for your applications, the same way RabbitMQ does? That way, both your application and rabbitmq have in common the same CA.

[sandhraprakash]

In our case, having same CA for RabbitMQ and client will not work, because the clients would be external and they would be providing their own CA. So If I have 5 clients, they may produce 5 different CAs which we were planning to bundle.

[MarcialRosales]

@sandhraprakash I totally understand your case. The certification path for server and client certificates do not share the same common root CA. Whereas the Vault support we have built in our k8s operator assumes that all certificates share a common root CA.

One suggestion I would like to make is the following (although I have not tried it). You can configure Vault to use intermediary CA to issue PKI for RabbitMQ (https://www.vaultproject.io/docs/secrets/pki#setting-up-intermediate-ca) that way you can control the root CA files the same way you have done up until now.

Once we configure RabbitMQ with Vault, it is assumed that everything related to secrets, including PKI, is managed thru Vault.

Please, let us know if the alternative suggested above works. If that did not work, there may be another alternative which would require code changes. We would have to specify vault paths from where to download more trusted CAs.

[sandhraprakash]

Before I tell which one would suit us, In the 2nd option,

1. Will the operator let us configure multiple vault paths as there will be multiple root CAs to be downloaded into the CA bundle?
2. When we were using K8 secret, dynamically updating K8 secret with a new CA bundle will immediately reflect in Rabbitmq without a restart or redeploy of rabbitmq. But in this case what would be the behaviour, if we add a new client with a new vault path, will we have to modify the rabbitmq.yaml and redeploy? How should this scenario be taken care of?

[MarcialRosales]

@sandhraprakash
For 1) and 2) Yes, that would be the sensible thing to do.

[sandhraprakash]

@MarcialRosales could you please help us understand the below 2 concerns with Vault approach

1. after the initial deployment, onboarding of new clients will create downtime for all clients, because we need to add a new vault path to rabbitmq deployment yaml and redeploy.
2. which type of vault engine will be expected of the path to be provided? If its pki engine as it is now, my understanding is pki engine requires a root CA and its private key for configuration in Vault, since we will be uploading root CA by client, no client will provide their private key.

[MarcialRosales]

@sandhraprakash We have not looked at how to implement it but your questions are very helpful in defining the requirements and the implementation. Thanks !

1. There are two ways of dealing with such situation. RabbitMQ could be configured with a Vault path which points to a list/folder where each entry/key is/contains either a) a root CA, or b) the location of a root CA in Vault. The latter is an indirection and it is more complex or less obvious. But it has the advantage that it allows you to keep your root CA in any location. Whereas with former, you have to place your root CAs together under a common root folder. Either options allows you to dynamically add root CAs. RabbitMQ would have to monitor this list though.
2. The engine would be a kv one. We do not need a pki as we do not want Vault to generate any pki material for us.

[sandhraprakash]

Vault would be a better approach than K8 secret in our case.
Thankyou @MarcialRosales for clarification. Eagerly waiting for the solution!

[sandhraprakash]

@MarcialRosales if this topic is already in your pipeline, could you please mention the ticket over here, in order to track it?

[MarcialRosales]

Hi @sandhraprakash, sorry for the delay. We are sharing with you a link to a ticket most likely this week. are you using the OpenSource cluster operator or the commercial one?

[sandhraprakash]

we are using the opensource cluster operator.

[HowardTwine]

Hi @sandhraprakash is this work for a Siemens specific project? I would like to know more and share our plans with RabbitMQ and MQTT. Is it possible to set up a call to discuss this please?

[sandhraprakash]

Sure @HowardTwine.