terrakube with keycloak authentication

[scabarrus]

Hello,

We try to use terrakube with dex configuration to use openid connect authentication via keycloak.

We use for each components an internal pki to provide certificate signed per our CA.

We import the certificate of our IdP in dex in folder /etc/ssl/certs and also in each other pod to see if it solve my issue.

We are able to authenticate with admin role, but when we try to create an organization, We have these errors:

• via the browser:
  GET https://terrakube-api.xxxxxx/api/v1/organization 500
• via pod output:
  Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

It seems the certificate need to be imported in keystore, but as I'm not fluent in Java, I would like to know if you have some guidance to help me.

Best regards,

[alfespa17]

Hello @scabarrus, let me check this during the weekend, there should be a way to add a new CA certs, we are basically using the spring buildpacks with the following maven command to create the docker images.

mvn spring-boot:build-image

I guess there should be an option inside the maven configuration to add the certs during the build. I will check this I think I can provide maybe a bash script that you can use to add the certs

[alfespa17]

Hello @scabarrus I was checking some documentation and it looks like there are two ways to add a custom certificate.

Add certificates binding at build time.

For this method I have updated the project configuration, you could include the certificates in PEM format inside the folder "bindings/ca-certificates" and create some custom images using the example build script.

AzBuilder/terrakube#265

Add certificates binding at runtime

To see how you can do it at runtime you can check this two examples:

Example1

Example2

[alfespa17]

I have updated the helm chart to support ca certificates at runtime, please use the following property security.caCerts

security:
  adminGroup: "TERRAKUBE_ADMIN"
  patSecret: "XXXXX"
  internalSecret: "XXXXXX"
  dexClientId: "microsoft"
  dexClientScope: "email openid profile offline_access groups"
  dexIssuerUri: "XXXXXX"
  caCerts:
    terrakubeDemo1.pem: |
      -----BEGIN CERTIFICATE-----
      
      XXX

      -----END CERTIFICATE-----
    terrakubeDemo2.pem: |
      -----BEGIN CERTIFICATE-----
      
      XXX

      -----END CERTIFICATE-----

Use this example to setup the ca certs inside the terrakube components, you will need to setup one env variable, a volume and volument mounts:

## API properties
api:
  enabled: true
  version: "2.7.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "terrakubeDemo1.pem"
          path: "terrakubeDemo1.pem"
        - key: "terrakubeDemo2.pem"
          path: "terrakubeDemo2.pem"
        - key: "type" #mandadory
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true
  properties:
    databaseType: "H2"
    

## Executor properties
executor:
  enabled: true
  version: "2.7.0"  
  replicaCount: "1"
  serviceType: "ClusterIP"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "terrakubeDemo1.pem"
          path: "terrakubeDemo1.pem"
        - key: "terrakubeDemo2.pem"
          path: "terrakubeDemo2.pem"
        - key: "type"
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true
  properties:
    toolsRepository: "https://github.com/AzBuilder/terrakube-extensions"
    toolsBranch: "main"

## Registry properties
registry:
  enabled: true
  version: "2.7.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "terrakubeDemo1.pem"
          path: "terrakubeDemo1.pem"
        - key: "terrakubeDemo2.pem"
          path: "terrakubeDemo2.pem"
        - key: "type"
          path: "type"
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true

If the configuration is correct the pods log will show something like:

[scabarrus]

Hi,

Thanks a lot, I'll try on Monday and give you a feedback.
If It works and you are interested, I can share with you the example with keycloak for people who wants to use dex with oidc authentication.

[alfespa17]

Sure, if you want to share the example feel free to send a PR to the repo to add the example configuration using OIDC

[scabarrus]

Hello,

Unfortunately something seems to not work from my side/
See below what I have done following your advices:

## API properties
api:
  enabled: true
  version: "2.7.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  resources: {}
  properties:
    databaseType: "H2"
    databaseHostname: ""
    databaseName: ""
    databaseUser: ""
    databasePassword: ""
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "my-pki-int.pem"
          path: "my-pki-int.pem"

  volumeMounts:
    - name: ca-certs
      mountPath: /mnt/platform/bindings/ca-certificates
      readOnly: true
executor:
  enabled: true
  version: "2.7.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  resources: {}
  properties:
    toolsRepository: "https://github.com/AzBuilder/terrakube-extensions"
    toolsBranch: "main"
    terraformStateType: ""
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "my-pki-int.pem"
          path: "my-pki-int.pem"

  volumeMounts:
    - name: ca-certs
      mountPath: /mnt/platform/bindings/ca-certificates
      readOnly: true

## Registry properties
registry:
  enabled: true
  version: "2.7.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  resources: {}
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "my-pki-int.pem"
          path: "my-pki-int.pem"

  volumeMounts:
    - name: ca-certs
      mountPath: /mnt/platform/bindings/ca-certificates
      readOnly: true

I get the following errors:

Enabling Java Native Memory Tracking
Adding 127 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=2 -XX:MaxDirectMemorySize=10M -Xmx5285361K -XX:MaxMetaspaceSize=231358K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.IllegalArgumentException: Cannot instantiate interface org.springframework.context.ApplicationListener : org.springframework.cloud.bindings.boot.BindingFlattenedEnvironmentPostProcessor
        at org.springframework.boot.SpringApplication.createSpringFactoriesInstances(SpringApplication.java:461)
        at org.springframework.boot.SpringApplication.getSpringFactoriesInstances(SpringApplication.java:443)
        at org.springframework.boot.SpringApplication.getSpringFactoriesInstances(SpringApplication.java:436)
        at org.springframework.boot.SpringApplication.<init>(SpringApplication.java:267)
        at org.springframework.boot.SpringApplication.<init>(SpringApplication.java:245)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
        at org.terrakube.api.ServerApplication.main(ServerApplication.java:18)
        ... 8 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.bindings.boot.BindingFlattenedEnvironmentPostProcessor]: Constructor threw exception; nested exception is java.lang.IllegalArgumentException: /mnt/platform/bindings/ca-certificates has no type and is not a valid binding

If I remove the env variable, I'm able to see the certificate (I trunc it) inside of the container:

cat /mnt/platform/bindings/ca-certificates/fabryk-pki-int.pem
-----BEGIN CERTIFICATE-----
MIIEOzCCAyOgAwIBAgIUPe7aQIjR2W15yHkfgM1FMvPmPZ8wDQYJKoZIhvcNAQEL
BQAwUzELMAkGA1UEBhMCRlIxEDAOBgNVBAoTB0t5bmRyeWwxCzAJBgNVBAsTAklU
...
...
-----END CERTIFICATE-----

If you have an idea ?

[alfespa17]

Hello @scabarrus, you need to add the key "type", this is a configuration file that needs to be included inside the directory where you have your certs in this case the folder /mnt/platform/bindings. this file is already included in the helm chart in the secret terrakube-ca-secrets

This file is already incluced inside the kubernetes secrets

apiVersion: v1
kind: Secret
metadata:
  name: terrakube-ca-secrets
type: Opaque
stringData: 
  type: |
    ca-certificates
  
  {{- with .Values.security.caCerts }}
  {{- toYaml . | nindent 2 }}
  {{- end }}

Reference: https://github.com/AzBuilder/terrakube-helm-chart/blob/main/templates/secrets-certs.yaml

This is the reason the error is showing:

IllegalArgumentException: /mnt/platform/bindings/ca-certificates has no type and is not a valid binding

Example:

## API properties
api:
  enabled: true
  version: "2.7.0"
  replicaCount: "1"
  serviceType: "ClusterIP"
  env:
  - name: SERVICE_BINDING_ROOT
    value: /mnt/platform/bindings
  volumes:
    - name: ca-certs
      secret:
        secretName: terrakube-ca-secrets
        items:
        - key: "terrakubeDemo1.pem"
          path: "terrakubeDemo1.pem"
        - key: "terrakubeDemo2.pem"
          path: "terrakubeDemo2.pem"
        - key: "type" # FILE REQUIRE TO ENABLE CUSTOM CERTS 
          path: "type" # FILE REQUIRE TO ENABLE CUSTOM CERTS 
  volumeMounts:
  - name: ca-certs
    mountPath: /mnt/platform/bindings/ca-certificates
    readOnly: true
  properties:
    databaseType: "H2"