Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

marathon and traefik 2.1 not reading /etc/hosts of container resulting in 504 Gateway timeout #6001

Closed
kopax opened this issue Dec 11, 2019 · 7 comments · May be fixed by containous/traefik-library-image#75
Closed

marathon and traefik 2.1 not reading /etc/hosts of container resulting in 504 Gateway timeout #6001

kopax opened this issue Dec 11, 2019 · 7 comments · May be fixed by containous/traefik-library-image#75
Labels
kind/question a question status/5-frozen-due-to-age

Comments

@kopax
Copy link

kopax commented Dec 11, 2019 •
edited
Loading

Bug

What did you do?

I have deployed the following app on marathon:

{
  "id": "/whoami",
  "cpus": 0.1,
  "mem": 256.0,
  "instances": 3,
  "labels": {
    "traefik.enable": "true",
    "traefik.http.routers.whoami.rule": "Host(`traefik-testing.mydomain.com`)",
    "traefik.http.routers.whoami.entrypoints": "web"
  },
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "containous/whoami",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 0,
          "name": "http-api",
          "protocol": "tcp"
        }
      ]
    }
  },
  "readinessChecks": [
    {
      "name": "readinessCheck",
      "protocol": "HTTP",
      "path": "/",
      "portName": "http-api",
      "intervalSeconds": 30,
      "timeoutSeconds": 10,
      "httpStatusCodesForReady": [200],
      "preserveLastResponse": false
    }
  ],
  "healthChecks": [
    {
      "path": "/",
      "portIndex": 0,
      "protocol": "HTTP",
      "gracePeriodSeconds": 30,
      "intervalSeconds": 10,
      "timeoutSeconds": 30,
      "maxConsecutiveFailures": 3
    }
  ]
}

This is the docker-compose.yml used to start the traefik container on localhost, that can reach marathon on the LAN through http://192.168.0.22:8080:

version: '3'
services:
  reverse-proxy:
    image: traefik:v2.1
    network_mode: "host"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./traefik.yaml:/etc/traefik/traefik.yaml"
      - "./letsencrypt:/letsencrypt"

What did you expect to see?

I expect to be able to succeed this command:

curl -H 'Host: traefik-testing.mydomain.com' -L http://localhost

What did you see instead?

Curl response:

Gateway Timeout

traefik debug log:

Output of traefik version: (What version of Traefik are you using?)

v2.1

What is your environment & configuration (arguments, toml, provider, platform, ...)?

This is my traefik.yaml

log:
  level: "DEBUG"
entryPoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
providers:
  marathon:
    endpoint: "http://192.168.0.22:8080"
    watch: true

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

reverse-proxy_1  | time="2019-12-11T13:45:56Z" level=debug msg="'504 Gateway Timeout' caused by: dial tcp 212.95.74.75:31400: i/o timeout"

Note: the IP it dial as nothing to do with my LAN, it is owned by my ISP and it is not even my WAN address.

On the dashboard, IP are correctly guessed and all are reachable by telnet from traefik host.

I am out of idea, it simply does not work with marathon, any idea what is failing here?

Thanks and best!
@kopax kopax changed the title 504 gateway timeout due ton wrong IP guessed by traefik marathon and traefik 2.1 always return 504 gateway timeout due ton wrong IP Dec 11, 2019
@kopax kopax mentioned this issue Dec 11, 2019
Merged
1 task
@kopax kopax changed the title marathon and traefik 2.1 always return 504 gateway timeout due ton wrong IP marathon and traefik 2.1 dial wrong IP and return 504 gateway timeout Dec 11, 2019
@traefiker
Copy link
Contributor

Hi! I'm Træfiker 🤖 the bot in charge of communication regulation.

Thanks for your interest in Traefik!

We dedicate the issue tracker to bug reports and feature requests only. My advanced AI has spotted that your issue might be a configuration problem or relates to something that doesn't look like a bug.

To confirm this, please join our Community Forum and reach out to us on the Traefik section.

In case I'm wrong (well, that would be embarrassing 😅), my developers will re-open the issue and fix me!

In the meantime, you can double check Traefik's documentation.

@kopax

This comment has been minimized.

Sign in to view
@kopax kopax mentioned this issue Dec 11, 2019
Closed
@kopax

This comment has been minimized.

Sign in to view
@kopax

This comment has been minimized.

Sign in to view
@kopax

This comment has been minimized.

Sign in to view
@kopax

This comment has been minimized.

Sign in to view
@kopax kopax changed the title marathon and traefik 2.1 dial wrong IP and return 504 gateway timeout marathon and traefik 2.1 not reading /etc/hosts of container resulting in 504 Gateway timeout Dec 12, 2019
@kopax
Copy link
Author

kopax commented Dec 12, 2019

Not related to #1243, it seems that /etc/hosts is ignored totally by traeffic.

Related to https://stackoverflow.com/questions/49476452/traefik-forwarding-to-a-host-and-overriding-ip

/etc/nsswitch.conf is absent from the system, doing:

echo "hosts: files dns" > /etc/nsswitch.conf

will solve the issue of /etc/hosts being ignore.

kopax added a commit to kopax/traefik that referenced this issue Dec 12, 2019
@kopax
fix(dns): /etc/hosts was ignored by traefik
7e80565 
Fix traefik#6001
@kopax kopax mentioned this issue Dec 12, 2019
Closed
kopax added a commit to kopax/traefik-library-image that referenced this issue Dec 12, 2019
@kopax
fix(dns): /etc/hosts was ignored by traefik
fe8e0c2 
Fix traefik/traefik#6001

### What does this PR do?

It create the missing `/etc/nsswitch.conf`.

### Motivation

During proxying, traefik is performing a DNS resolution. This DNS resolution is skipping `/etc/hosts` file. This can cause unwanted DNS resolution, especially when using hostname as fqdn in marathon.

If you use `/etc/hosts` file for some resolution, this will fix it. If you use a real fqdn in marathon on a private network, you will still have to properly configure DNS serveur in traefik host.
@kopax kopax mentioned this issue Dec 12, 2019
Open
@traefik traefik locked and limited conversation to collaborators Jan 15, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/question a question status/5-frozen-due-to-age
Projects
None yet
Milestone
No milestone
2 participants
@kopax @traefiker