TLS handshake error

[ralphtheninja]

What version of Traefik are you using (traefik version)?

1.1.2

What is your environment & configuration (arguments, toml...)?

Running on Azure using Azure Load Balancer in front of two manager nodes. Running traefik on one manager.

Running traefik in docker swarm mode with command

docker service create --name traefik \
--constraint 'node.role == manager' \
--network traefik-net -p 80:80 -p 443:443 -p 8080:8080 \
--mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock,ro=true \
--mount type=bind,src=/path/to/traefik.toml,dst=/traefik.toml,ro=true \
--mount type=bind,src=/path/to/acme,dst=/etc/traefik/acme,ro=false \
traefik:v1.1.2

traefik.toml is standard with swarmmode set to true

What did you do?

Started the service

What did you expect to see?

No TLS errors

What did you see instead?

I get numerous of the following messages (every 30 seconds or so).

2017/01/07 12:50:25 server.go:2317: http: TLS handshake error from \
10.255.0.4:3416: read tcp 10.255.0.5:443->10.255.0.4:3416: read: \
connection reset by peer

As a side note, I have ondemand set to true and succeed in getting the ACME certificate:

time="2017-01-07T12:30:14Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:0xc420325350 Auth:<nil> Compress:false}" 
time="2017-01-07T12:30:14Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc4201b1980 Redirect:<nil> Auth:<nil> Compress:false}" 
time="2017-01-07T12:30:14Z" level=info msg="Starting server on :80" 
time="2017-01-07T12:30:15Z" level=warning msg="ACME.StorageFile is deprecated, use ACME.Storage instead" 
time="2017-01-07T12:30:15Z" level=info msg="Generating ACME Account..." 
time="2017-01-07T12:30:18Z" level=debug msg="Building ACME client..." 
time="2017-01-07T12:30:18Z" level=info msg=Register... 
time="2017-01-07T12:30:19Z" level=debug msg=AgreeToTOS... 
time="2017-01-07T12:30:19Z" level=info msg="Starting provider *provider.Docker {\"Watch\":true,\"Filename\":\"\",\"Constraints\":null,\"Endpoint\":\"unix:///var/run/docker.sock\",\"Domain\":\
"REDACTED.com\",\"TLS\":null,\"ExposedByDefault\":true,\"UseBindPortIP\":false,\"SwarmMode\":true}" 
time="2017-01-07T12:30:19Z" level=info msg="Starting provider *main.WebProvider {\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false,\"Auth\":null}" 
time="2017-01-07T12:30:19Z" level=info msg="Retrieving ACME certificates..." 
time="2017-01-07T12:30:19Z" level=debug msg="Loading ACME certificates [REDACTED.com]..." 
time="2017-01-07T12:30:19Z" level=info msg="Starting server on :443" 
time="2017-01-07T12:30:19Z" level=debug msg="Docker connection established with docker 1.13.0-rc5 (API 1.25)" 
time="2017-01-07T12:30:19Z" level=debug msg="Filtering container without port and no traefik.port label traefik" 
time="2017-01-07T12:30:19Z" level=debug msg="Configuration received from provider docker: {}" 
time="2017-01-07T12:30:19Z" level=debug msg="Last docker config received more than 2s, OK" 
time="2017-01-07T12:30:19Z" level=info msg="Server configuration reloaded on :80" 
time="2017-01-07T12:30:19Z" level=info msg="Server configuration reloaded on :443" 
time="2017-01-07T12:30:20Z" level=debug msg="Challenge Present REDACTED.com" 
time="2017-01-07T12:30:21Z" level=debug msg="Challenge GetCertificate 3bb03cc32185ea2403220ebf82ad439b.77e067e113f9e50b16dcbd94cd914382.acme.invalid" 
time="2017-01-07T12:30:21Z" level=debug msg="ACME got challenge 3bb03cc32185ea2403220ebf82ad439b.77e067e113f9e50b16dcbd94cd914382.acme.invalid" 
time="2017-01-07T12:30:21Z" level=debug msg="Challenge CleanUp REDACTED.com" 
time="2017-01-07T12:30:27Z" level=debug msg="Loaded ACME certificates [REDACTED.com]" 

[ralphtheninja]

I tried checking the code in server.go but it seems to me that it's not the server.go file from traefik (it doesn't have the same format as log output from traefik), also line 2317 is too large.

[ralphtheninja]

Also port 3416 seem to vary. In another setup I now get

2017/01/08 13:08:44 server.go:2317: http: TLS handshake error from 10.255.0.3:1504: read tcp 10.255.0.4:443->10.255.0.3:1504: read: connection reset by peer

[ralphtheninja]

After some more investigation and snooping around I could deduce that 10.255.0.3 comes from the ingress network:

$ docker inspect network ingress
[
    {
        "Name": "ingress",
        "Id": "zo68p55cvxo8pt6j5scf9sfcr",
        "Created": "2017-01-08T12:38:41.3815157Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.255.0.0/16",
                    "Gateway": "10.255.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "9e1a38a89a28a2145cbe6be52da61a6fe55f900b69a1620d722925cb414d5450": {
                "Name": "traefik.1.o9ff835y1he90mfxvcirne01k",
                "EndpointID": "0f9b97180d95e4ec89de822c03dfccaecca01ca13ce6313775be7cd7fddfae99",
                "MacAddress": "02:42:0a:ff:00:04",
                "IPv4Address": "10.255.0.4/16",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "9f7fad26707b892ddc138fb81274c9407e1f0476a6dc436bc68b254a28d460fc",
                "MacAddress": "02:42:0a:ff:00:03",
                "IPv4Address": "10.255.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "kalejdo-manager-1-fa66ff43304f",
                "IP": "10.1.2.4"
            }
        ]
    }
]

[dtomcej]

@ralphtheninja I see similar issues in kubernetes. Connections in kubernetes are showing tcp connections to IPs that are non-existent, but are attributed to the same line in the same file.

Currently it doesn't affect operations, and can be ignored. I will be looking into it at a later date.

[ralphtheninja]

@dtomcej Ok! At first I thought it was something going on with Azure and the load balancer. I'm guessing something is trying to connect to traefik over tls via the ingress endpoint?

[ralphtheninja]

@dtomcej I know what this is about now. It's from one of the probes in the azure load balancer.

[sedouard]

Seeing this same issue on 1.12 on Docker Swarm on Azure (although I only see it reproduce on a Traefik instance sitting across a VPN).

For us, the IP address referrred to exists on the other side of our VPN. Enter containers on the same local network as traefik shows that the IP resolves. However we still hit this error within the traefik container. Restarting the container resolves it