Handle Custom Domain with SSL on Fabio

[maddy023]

Problem
Currently Nomad Cluster is deployed along with fabio -> AWS Application Load Balancer.
Where Load Balancer is configured with domain1.com SSL cert with ACM , where all the job deployed in nomad cluster with *.domain1.com works as expected., where route53 records are created for respective jobs.

When moving on with custom domain where as helo.domain2.com , i did map the route53 records to reroute the traffic but i am facing SSL cert issue here as load balancer is configure with different ACM.

Test Cases:
Deployed NLB and mapped nomad cluster to handle the traffic , but this case how to handle multiple custom domains? Need to deploy fabio for all the custom domains which are setup ? also how to handle the traffic with SSL Certs (lets say used letsencrypt to generate CA certs )

Is there any feature or work around in fabio to handle multiple custom certs to serve traffic with SSL Certs?

[tristanmorgan]

To handle multiple domain names on Fabio you can use tags on the consul services like"urlprefix-helo.domain2.com/"and then you just need to work out how you want to setup your certificate store (proxy.cs configuration).

[maddy023]

fabio.properties
proxy.cs = cs=consul;type=consul;cert=http://localhost:8500/v1/kv/ssl
proxy.addr = :9999;proto=http,:9995;proto=https;cs=consul

Used letsencrypt to generate certs for domain and placed in consul kv :
cat fullchain.pem privkey.pem > cert01.pem

as per : #621

Cluster as nomad + Consul Deployment - deployed loki
`

job "loki" {
  datacenters = ["us-west-1"]
  type        = "service"

  constraint {
    attribute = meta.cluster
    operator  = "="
    value     = "proxy"
  }

  group "loki" {
    count = 1

    network {

      port "http" {
        to = 3100
      }
    }

    restart {
      attempts = 3
      delay    = "20s"
      mode     = "delay"
    }

    task "loki" {
      driver = "docker"

      config {
        image = "grafana/loki:2.6.1"
        ports = ["http"]

      resources {
        cpu    = 100
        memory = 100
      }

      service {
        name = "loki"
        port = "http"
        tags = ["urlprefix-devops.helper.co/"]
      }
    }
  }
}

Fabio Logs:

`

sh-4.2$ ./fabioex -cfg ./fabio/fabio.properties
2023/02/28 09:53:07 [INFO] Setting log level to INFO
2023/02/28 09:53:07 [INFO] Runtime config
{
    "Proxy": {
        "Strategy": "rnd",
        "Matcher": "prefix",
        "NoRouteStatus": 404,
        "MaxConn": 10000,
        "ShutdownWait": 0,
        "DeregisterGracePeriod": 0,
        "DialTimeout": 30000000000,
        "ResponseHeaderTimeout": 0,
        "KeepAliveTimeout": 0,
        "IdleConnTimeout": 15000000000,
        "FlushInterval": 1000000000,
        "GlobalFlushInterval": 0,
        "LocalIP": "198.54.12.11",
        "ClientIPHeader": "",
        "TLSHeader": "",
        "TLSHeaderValue": "",
        "GZIPContentTypes": null,
        "RequestID": "",
        "STSHeader": {
            "MaxAge": 0,
            "Subdomains": false,
            "Preload": false
        },
        "AuthSchemes": {},
        "GRPCMaxRxMsgSize": 4194304,
        "GRPCMaxTxMsgSize": 4194304
    },
    "Registry": {
        "Backend": "consul",
        "Static": {
            "NoRouteHTML": "",
            "Routes": ""
        },
        "File": {
            "NoRouteHTMLPath": "",
            "RoutesPath": ""
        },
        "Consul": {
            "Addr": "localhost:8500",
            "Scheme": "http",
            "Token": "",
            "KVPath": "/fabio/config",
            "NoRouteHTMLPath": "/fabio/noroute.html",
            "TagPrefix": "urlprefix-",
            "Register": true,
            "ServiceAddr": ":9998",
            "ServiceName": "fabio",
            "ServiceTags": null,
            "ServiceStatus": [
                "passing"
            ],
            "CheckInterval": 1000000000,
            "CheckTimeout": 3000000000,
            "CheckScheme": "http",
            "CheckTLSSkipVerify": false,
            "ChecksRequired": "one",
            "ServiceMonitors": 1,
            "TLS": {
                "KeyFile": "",
                "CertFile": "",
                "CAFile": "",
                "CAPath": "",
                "InsecureSkipVerify": false
            },
            "PollInterval": 0,
            "RequireConsistent": true,
            "AllowStale": false
        },
        "Custom": {
            "Host": "",
            "Path": "",
            "QueryParams": "",
            "Scheme": "https",
            "CheckTLSSkipVerify": false,
            "PollInterval": 5,
            "NoRouteHTML": "",
            "Timeout": 10
        },
        "Timeout": 10000000000,
        "Retry": 500000000
    },
    "Listen": [
        {
            "Addr": ":9999",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "IdleTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null,
                "VaultFetchToken": ""
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null,
            "ProxyProto": false,
            "ProxyHeaderTimeout": 0,
            "Refresh": 0
        },
        {
            "Addr": ":9995",
            "Proto": "https",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "IdleTimeout": 0,
            "CertSource": {
                "Name": "consul",
                "Type": "consul",
                "CertPath": "http://localhost:8500/v1/kv/ssl",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null,
                "VaultFetchToken": ""
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null,
            "ProxyProto": false,
            "ProxyHeaderTimeout": 0,
            "Refresh": 0
        }
    ],
    "Log": {
        "AccessFormat": "common",
        "AccessTarget": "",
        "RoutesFormat": "delta",
        "Level": "INFO"
    },
    "Metrics": {
        "Target": "",
        "Prefix": "{{clean .Hostname}}.{{clean .Exec}}",
        "Names": "{{clean .Service}}.{{clean .Host}}.{{clean .Path}}.{{clean .TargetURL.Host}}",
        "Interval": 30000000000,
        "Timeout": 10000000000,
        "Retry": 500000000,
        "GraphiteAddr": "",
        "StatsDAddr": "",
        "DogstatsdAddr": "",
        "Circonus": {
            "APIKey": "",
            "APIApp": "fabio",
            "APIURL": "",
            "CheckID": "",
            "BrokerID": "",
            "SubmissionURL": ""
        },
        "Prometheus": {
            "Subsystem": "",
            "Path": "/metrics",
            "Buckets": [
                0.005,
                0.01,
                0.025,
                0.05,
                0.1,
                0.25,
                0.5,
                1,
                2.5,
                5,
                10
            ]
        }
    },
    "UI": {
        "Listen": {
            "Addr": ":9998",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "IdleTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "Header": null,
                "VaultFetchToken": ""
            },
            "StrictMatch": false,
            "TLSMinVersion": 0,
            "TLSMaxVersion": 0,
            "TLSCiphers": null,
            "ProxyProto": false,
            "ProxyHeaderTimeout": 0,
            "Refresh": 0
        },
        "Color": "light-green",
        "Title": "",
        "Access": "rw",
        "RoutingTable": {
            "Source": {
                "LinkEnabled": false,
                "NewTab": true,
                "Scheme": "http",
                "Host": "",
                "Port": ""
            }
        }
    },
    "Runtime": {
        "GOGC": 100,
        "GOMAXPROCS": 2
    },
    "Tracing": {
        "TracingEnabled": false,
        "CollectorType": "http",
        "ConnectString": "http://localhost:9411/api/v1/spans",
        "ServiceName": "Fabiolb",
        "Topic": "Fabiolb-Kafka-Topic",
        "SamplerRate": -1,
        "SpanHost": "localhost:9998",
        "SpanName": "",
        "TraceID128Bit": true
    },
    "ProfileMode": "",
    "ProfilePath": "/tmp",
    "Insecure": false,
    "GlobMatchingDisabled": false,
    "GlobCacheSize": 1000,
    "BGP": {
        "BGPEnabled": false,
        "Asn": 65000,
        "AnycastAddresses": null,
        "RouterID": "",
        "ListenPort": 179,
        "ListenAddresses": [
            "0.0.0.0"
        ],
        "Peers": null,
        "EnableGRPC": false,
        "GRPCListenAddress": "127.0.0.1:50051",
        "GRPCTLS": false,
        "CertFile": "",
        "KeyFile": "",
        "GOBGPDCfgFile": "",
        "NextHop": ""
    }
}
2023/02/28 09:53:07 [INFO] Version 1.6.3 starting
2023/02/28 09:53:07 [INFO] Go runtime is go1.19.4
2023/02/28 09:53:07 [INFO] Registering metrics provider ""
2023/02/28 09:53:07 [INFO] Metrics disabled
2023/02/28 09:53:07 [INFO] Setting GOGC=100
2023/02/28 09:53:07 [INFO] Setting GOMAXPROCS=2
2023/02/28 09:53:07 [INFO] consul: Connecting to "localhost:8500" in datacenter "us-west-1"
2023/02/28 09:53:07 [INFO] Admin server access mode "rw"
2023/02/28 09:53:07 [INFO] Admin server listening on ":9998"
2023/02/28 09:53:07 [INFO] Waiting for first routing table
2023/02/28 09:53:07 [INFO] consul: Using dynamic routes
2023/02/28 09:53:07 [INFO] consul: Using tag prefix "urlprefix-"
2023/02/28 09:53:07 [INFO] consul: Watching KV path "/fabio/config"
2023/02/28 09:53:07 [INFO] consul: Watching KV path "/fabio/noroute.html"
2023/02/28 09:53:07 [INFO] HTTP proxy listening on :9999
2023/02/28 09:53:07 [INFO] HTTPS proxy listening on :9995
2023/02/28 09:53:07 [INFO] Access logging disabled
2023/02/28 09:53:07 [INFO] Using routing strategy "rnd"
2023/02/28 09:53:07 [INFO] Using route matching "prefix"
2023/02/28 09:53:07 [INFO] Access logging disabled
2023/02/28 09:53:07 [INFO] Using routing strategy "rnd"
2023/02/28 09:53:07 [INFO] Using route matching "prefix"
2023/02/28 09:53:07 [INFO] cert: Store has certificates for ["devops.helper.co"]
2023/02/28 09:53:07 [INFO] Config updates
+ route add loki devops.helper.co/ http://198.54.12.11:31454/
2023/02/28 09:53:07 [INFO] consul: Registered fabio as "fabio"
2023/02/28 09:53:07 [INFO] consul: Registered fabio with id "fabio-ip-198.54.12.11.us-west-1.compute.internal-9998"
2023/02/28 09:53:07 [INFO] consul: Registered fabio with address "198.54.12.11"
2023/02/28 09:53:07 [INFO] consul: Registered fabio with tags ""
2023/02/28 09:53:07 [INFO] consul: Registered fabio with check &{CheckID:fabio-ip-198.54.12.11.us-west-1.compute.internal-9998-ttl Name: Args:[] DockerContainerID: Shell: Interval: Timeout: TTL:15s HTTP: Header:map[] Method: Body: TCP: Status: Notes: TLSServerName: TLSSkipVerify:false GRPC: GRPCUseTLS:false H2PING: H2PingUseTLS:false AliasNode: AliasService: SuccessBeforePassing:0 FailuresBeforeWarning:0 FailuresBeforeCritical:0 DeregisterCriticalServiceAfter:1m0s}
2023/02/28 09:53:07 [INFO] consul: Registered fabio with check &{CheckID: Name: Args:[] DockerContainerID: Shell: Interval:1s Timeout:3s TTL: HTTP:http://[198.54.12.11]:9998/health Header:map[] Method: Body: TCP: Status: Notes: TLSServerName: TLSSkipVerify:false GRPC: GRPCUseTLS:false H2PING: H2PingUseTLS:false AliasNode: AliasService: SuccessBeforePassing:0 FailuresBeforeWarning:0 FailuresBeforeCritical:0 DeregisterCriticalServiceAfter:}
`

When i hit the Server IP or domain : https://devops.helper.co

[tristanmorgan]

Have you seen any errors in the Fabio logs saying the connection failed? I think it might be the NLB or the SG's on the machine not letting traffic in. (with NLB's you need to add rules on the ec2 instances to accept the internet based traffic).

[maddy023]

Yeah , there is no logs in fabio.
also i am trying to connect via machine public ip, removed NLB out of picture for now.

mapped machine ip to the domain , but still same.

[tristanmorgan]

When you have the Fabio listener is on 9998 and you try to directly connect then you'll need to add the port to the urlprefix "urlprefix-devops.helper.co:9998/". alternatives are to listen on 443 with CAP_NET_BIND_SERVICE or maybe use some Iptables magic to nat the traffic.
essentially if its a Fabio config error I'd expect to see some sort of error message. maybe check the traffic is arriving with tcpdump...