Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot get multi-path working #1120

Closed
erik777 opened this issue Aug 12, 2017 · 28 comments
Closed

Cannot get multi-path working #1120

erik777 opened this issue Aug 12, 2017 · 28 comments

Comments

@erik777
Copy link

erik777 commented Aug 12, 2017

I have tried every configuration in the universe.

"work": Can access all URLs without an error in multiple browsers and with curl.

"does not work": Sends back 404 or 502 errors. If you refresh on some URLs with some configs, it can toggle between 3 different error results, two are different 404s.

I have two backend services. They work consistently with port-forward, LoadBalance in the service, and basic ingress without paths.

I get the root to map, but not any service below that in the path.

What I want is "/" to route to a node app in nginx and "/service2" and subsequent services to route to various services used by that app. I couldn't even get "/service1" and "/service2" working without the root path, which matched the examples. I tried following every example, then experimented into the unknown. No matter what I did, no path other than the root one would map successfully to a service.

I am running K8S 1.6.7 on GKE. Having ingress.kubernetes.io/rewrite-target: / doesn't help. When these errors occur, it is simply not getting to the service.

When I am inside a pod, I can curl port 80 of a service no problem. curl -v http://myservice returns the expected 200. When I get a 200, on the node app, I get a log from nginx.

Note that I did not create any controllers. I am just using gce. Nevertheless, I tried adding kubernetes.io/ingress.class: "gce" with no effect.

I spent over 8 hours trying every possibility I could think of. I deleted and re-created the ingress probably 100-200 times. I even deleted and re-created the cluster.

Going back to the original configuration, where you cannot access either service via the Ingress LB, you consistently get 404 when trying to access the service. I have noticed it is slightly different when I include a slash at the end. Without it, it returns this to curl:

< Content-Length: 0
< Date: Sat, 12 Aug 2017 17:42:38 GMT
< Via: 1.1 google

When I add a slash, I get that with the content default backend - 404. E.g.,

curl -v http://host/collector/

Here is the original multi-path config:

kind: Ingress
metadata:
  name: ingress-combined
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.global-static-ip-name: sc-static-ip
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: sc-admin
          servicePort: 80
      - path: /collector
        backend:
          serviceName: sc-collector
          servicePort: 80

This basic config works no matter which service I point it to:

kind: Ingress
metadata:
  name: basic-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: sc-static-ip
spec:
  backend:
    serviceName: sc-admin
    servicePort: 80

It is only multi-path with anything other than path: / that does not work.

Thank you in advance for your help.

@erik777
Copy link
Author

erik777 commented Aug 12, 2017

In an interesting twist, I switch the services. It invokes both services for the path root. But:

  • Returns default backend - 404 if I then try to access anything other than the service root.
  • includes the path instead of doing a re-write, ignoring the annotation ingress.kubernetes.io/rewrite-target: /.

I could work around the second problem by giving each service a context path if the health checks didn't require a 200 from the root.

@aledbf
Copy link
Member

aledbf commented Aug 12, 2017

I am running K8S 1.6.7 on GKE. Having ingress.kubernetes.io/rewrite-target: / doesn't help. When these errors occur, it is simply not getting to the service.

By default you get a GCE ingress controller running. Because of this:

  • kubernetes.io/ingress.class: "gce" annotation has no effect
  • ingress.kubernetes.io/rewrite-target annotation is not supported by the GCE ingress controller

@erik777
Copy link
Author

erik777 commented Aug 12, 2017

I found a workaround. Using these paths works:

  - path: /*
  - path: /test/*

Note that http://host/test still resolves to the first service, whereas http://host/test/ resolves to the second service. The services don't have to be human friendly URLs, so that is not an issue.

I clicked on every link in the Node app and couldn't locate a single 404! YAY! curl seemed happy, too.

The question I have is how come I couldn't find an example using a wildcard in the path in all the documentation on Ingress? The only reason I thought of it was because I figured out that GKE was using GCE URL Maps under the hood, and learned that it supports wild cards.

I did locate these discussions, which leave this as an open issue.

It has just been a long 10+ hour confusing ride due to documentation not being clear on this. And, to be sure, Google is #1 to earn credit here because I started out with and learned about Ingress from their GKE Setting up HTTP Load Balancing with Ingress documentation where their fanout example shows a root and a context under it pointing to two services WITHOUT asterisks! That was the example I started with, and the config I have been trying to get working. They really do need to update that page, and explain the impact of having or not having a wild card.

Hopefully, someone finds this issue in the first hour of their Ingress adventure. :)

@aledbf aledbf closed this as completed Aug 30, 2017
@anderfsilva
Copy link

I am facing the same issue using Nginx Ingress on AWS. Unfortunately it took me 10+ hours to find this thread. :(

@ryan-flores
Copy link

I sorted out by adding nginx.ingress.kubernetes.io/rewrite-target annotation. I'm using LetsEncrypt and not allowing http, but https.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
namespace: default
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
ingress.kubernetes.io/service-upstream: "true"
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /

@mleneveut
Copy link

mleneveut commented Jun 27, 2018

I have a similar issue with Azure AKS (v1.9.6) and a nginx (nginx-ingress-controller-0.15.0) :

I want to have different paths to route requests to different backends.

The following configuration works, but when I request "https://xxx/iv-da/swagger", the /swagger/v0/swagger.json AJAX call makes a request to "https://xxx/swagger/v0/swagger.json" without the "iv-da" context.

How could it keep the context for each backend's inside requests ?

{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "k8s-ingress",
    "namespace": "default",
    "annotations": {
      "certmanager.k8s.io/cluster-issuer": "letsencrypt-prod",
      "kubernetes.io/ingress.class": "nginx",
      "nginx.ingress.kubernetes.io/rewrite-target": "/"
    }
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "xxx"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "xxx",
        "http": {
          "paths": [
            {
              "path": "/iv-da",
              "backend": {
                "serviceName": "iv-da",
                "servicePort": 80
              }
            },
            {
              "path": "/",
              "backend": {
                "serviceName": "config-server",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

@crsantini
Copy link

crsantini commented Jul 23, 2018

In my case, I had two services, one running at / and another at /ui . I tried several combinations and the only one that worked with K8s 1.10 was

http://mydomain.com/*
Service 1 Ingress
kubernetes.io/ingress.class: "nginx"
path: /

http://mydonain.com/ui/app/index.html
Service 2 Ingress
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /ui
path: /ui/*

Note that for service 2, my tomcat was also initialising at /ui , so the rewrite-target did the trick otherwise, I had to add two "ui" to my url like http://mydonain.com/ui/ui/app/index.html

@lemonshow
Copy link

lemonshow commented Aug 9, 2018

@erik777
I got the same issue. The base works but the multi-path not. I am using Nginx ingress controller + AWS NLB and try to bring up my two services with the Ingress. It always routes to default and I am getting 404. Anyone used with AWS EKS with luck? Use rewrite did not help either. Any other suggestion? thanks,

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
namespace: local
annotations:
ingress.kubernetes.io/rewrite-target: '/'
spec:
#backend:
#serviceName: app1-service
#servicePort: 80
rules:

  • http:
    paths:
    • backend:
      serviceName: app1-service
      servicePort: 80
      path: /*
    • backend:
      serviceName: app2-service
      servicePort: 80
      path: /app2/*

@aledbf
Copy link
Member

aledbf commented Aug 9, 2018

@lemonshow remove * from the paths

@lemonshow
Copy link

It works now!
Actually I had another issue after this where I was testing with NLB but I comment out https.
I saw 308 redirect to https. Uncomment https, things start to work.
thanks a lot,

@lemonshow
Copy link

Maybe I missed somewhere, why I can see the default backend status(I have 2 instances in 2 zones) showing one healthy, one unhealthy in the AWS NLB?

@dolphub
Copy link

dolphub commented Aug 13, 2018

@mleneveut Did you figure out your issue yet? I am also interested in this. I'm serving a web application with socket connections and am unable to get socket.io requests to persist the proper url after using the rewrite target annotation.

@mleneveut
Copy link

@dolphub I ended up doing this :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: myingress
spec:
  rules:
  - host: myhost.com
    http:
      paths:
      - backend:
          serviceName: api
          servicePort: 80
        path: /domain-api
      - backend:
          serviceName: bff
          servicePort: 80
        path: /api
      - backend:
          serviceName: front
          servicePort: 80
        path: /
  tls:
  - hosts:
    - myhost.com
    secretName: tls-secret

And for the swagger problem, it is inside the code that we need to force the context, it is not an ingress problem.

Hope it helps.

@ryanQL
Copy link

ryanQL commented Sep 3, 2018

Hi, I have the Nginx Ingress running without issues in Production and HTTPS by using LetsEncrypt certificates:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: default
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/service-upstream: "true"
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.allow-http: "false"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: "1M"
spec:
  tls:
    - hosts:
      - developer.test.com
      secretName: developer-test-com-tls
    - hosts:
      - broker.test.com
      secretName: broker-test-com-tls
    - hosts:
      - dashboard.test.com
      secretName: dashboard-test-com-tls
  rules:
    - host: developer.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: developer-docs-service
              servicePort: 8080
          - path: /api
            backend:
              serviceName: developer-service
              servicePort: 8080
    - host: broker.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: broker-rabbitmq-service
              servicePort: 15672
    - host: dashboard.test.com
      http:
        paths:
          - path: /
            backend:
              serviceName: dashboard-service
              servicePort: 80

I set up a quick repo with the resources I used for the production environment. You need to replace the "test" subdomain with your own and set up the SSL certificates.

Hope it helps.

@nimendra
Copy link

I sorted out by adding nginx.ingress.kubernetes.io/rewrite-target annotation. I'm using LetsEncrypt and not allowing http, but https.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
namespace: default
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
ingress.kubernetes.io/service-upstream: "true"
kubernetes.io/ingress.class: "nginx"
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /

to make matter worse for me, I had
ingress.kubernetes.io/rewrite-target: / instead, which was hard to spot!

@gpflaum
Copy link

gpflaum commented Feb 4, 2019

@nimendra Thank you for leaving that comment! We too had this, which was working until we upgraded from an old ingress-nginx version (0.9.0-beta.11 -> 0.22.0):

ingress.kubernetes.io/rewrite-target: /

After many hours, I found your comment and changed to this, which works:

nginx.ingress.kubernetes.io/rewrite-target: /

@dillilabs
Copy link

Refer to Step 6 : Serving Multiple Applications on a Load Balancer on this Google Kubernetes Engine tutorial on Setting up HTTP Load Balancing with Ingress. It exactly answers the questions raised in this thread.

To route path / to backend service web on port 8080 and path /v2 to backend service web2 on port 8080 , the spec section of the ingress yaml file will be:

spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: web
          servicePort: 8080
      - path: /v2/*
        backend:
          serviceName: web2
          servicePort: 8080

Ensure that backend services respond with a 200OK for /request. Note that web2 needs to respond with a 200OK to a root path / ( not /v2/). This is important for Google Load Balancer to perform a successful health check on the two services before making them available. If they do not pass the health checks, you would get a 502 Error.

In our case, we have a angular frontend service with an express backend. We want to angular to serve / and express to serve /api for the same host.

So, in our express request handling file, we have the following added to respond with a 200OK for path /.

const app = express();
app.use('/', function (req, res, next) {
	res.sendStatus(200);
	next();
});

Hope this helps.

@vasudev-hv
Copy link

vasudev-hv commented May 9, 2019

Ingress version 0.22.0 or higher has changed the way how rewrite-target works.
You'll need to regex-match the path and add it to the rewrite-target.
nginx.ingress.kubernetes.io/rewrite-target: /$2 and

  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

Refer to changelog here
How to article here

@felipecrescencio
Copy link

In my case, I based my solution in this tutorial and I have the following:

  1. FRONTEND: React Application
    • Path: /
    • Full URL doesn't matter for application
  2. BACKEND
    • Path: /api
    • Full URL MATTERS for application
  3. I am using GCP

The only way it worked for me was creating 2 ingress, because
If I use nginx.ingress.kubernetes.io/rewrite-target: / It breaks my backend that needs full url path

If I use nginx.ingress.kubernetes.io/rewrite-target: /$2 It breaks my frontend, showing a blank home page.

My solution:

  1. FRONTEND Ingress:
nginx.ingress.kubernetes.io/rewrite-target: /
path: /
  1. BACKEND Ingress:
nginx.ingress.kubernetes.io/rewrite-target: /$2
path: /api(/|$)(.*)

I attached full ingress file yaml.

Hope it helps and thank you so much, this thread helped me a lot!

@LorenzoR
Copy link

@felipecrescencio if you create 2 ingress, do you have to pay for 2 load balancers in GCP?

@aledbf
Copy link
Member

aledbf commented May 16, 2019

if you create 2 ingress, do you have to pay for 2 load balancers in GCP?

@LorenzoR no with ingress-nginx

@demisx
Copy link

demisx commented Aug 2, 2019

If you are using v0.22+, you need to use capture groups. https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-rules
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - dl-web.stg.myhost.org
      secretName: letsencrypt-prod
  rules:
    - host: dl-web.stg.myhost.org
      http:
        paths:
          - path: /
            backend:
              serviceName: dl-web
              servicePort: 80
          - path: /api(/|$)(.*)
            backend:
              serviceName: dl-api
              servicePort: 3000

@aditanix
Copy link

I as well can confirm that I spent upwards of 8 hours on the same issue. Using /* instead of / was the solution.

@DaveSlinn
Copy link

DaveSlinn commented Nov 7, 2019

Everyone is saying that using /* or /api/* works, but it's not working for me. This is my ingress def:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hmc-ingress
  namespace: hmc
spec:
  tls:
  - hosts: 
    - hmc.gms.ca
    secretName: ssl-secret
  rules:
  - host: hmc.gms.ca
    http:
      paths:
      - path: /hmc-shell/*
        backend:
          serviceName: hmc-shell-service
          servicePort: 80

when I go to https://hmc.gms.ca/hmc-shell I end up getting the default http backend and not the service running on hmc-shell-service.
When I remove the /* from the path to - path: /hmc-shell, I manage to download the default doc from my hmc-shell-service, but any subsequent request for other files, like css and js, don't return the proper content. They each just resend the default doc content.

Argh!! What am I doing wrong?

Update: I've tried pouring over the nginx documentation, and using a regex:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hmc-ingress
  namespace: hmc
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1$2$3
spec:
  tls:
  - hosts: 
    - hmc.gms.ca
    secretName: ssl-secret
  rules:
  - host: hmc.gms.ca
    http:
      paths:
      - path: /(hmc-shell)(/|$)(.*)
        backend:
          serviceName: hmc-shell-service
          servicePort: 80

gives me this really weird nginx.conf:

server {
	server_name hmc.gms.ca ;
	
	listen 80;
	
	listen [::]:80;
	
	set $proxy_upstream_name "-";
	
	listen 443  ssl http2;
	
	listen [::]:443  ssl http2;
	
	# PEM sha: 81da239514bc48d4f2c5497126606f4289d4a2c2
	ssl_certificate                         /etc/ingress-controller/ssl/hmc-ssl-secret.pem;
	ssl_certificate_key                     /etc/ingress-controller/ssl/hmc-ssl-secret.pem;
	
	location ~* "^/(hmc-shell)(/|$)(.*)" {
		
		set $namespace      "hmc";
		set $ingress_name   "hmc-ingress";
		set $service_name   "hmc-shell-service";
		set $service_port   "80";
		set $location_path  "/(hmc-shell)(/|${literal_dollar})(.*)";	

What's with the ${literal_dollar} in the location path? Is that an nginx thing?

@JorgeGuerreroTech
Copy link

Ingress version 0.22.0 or higher has changed the way how rewrite-target works.
You'll need to regex-match the path and add it to the rewrite-target.
nginx.ingress.kubernetes.io/rewrite-target: /$2 and

  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)

Refer to changelog here
How to article here

Yesss! This worked for me Thanks you!!

@bedeabza
Copy link

I will add my two cents here, even though my issue was a "user problem", but in case anyone generates their YAML using envsubst they might encounter this which took me a while to find and fix.

When using nginx.ingress.kubernetes.io/rewrite-target: /$1 the last part $1 will get replaced by envsubst most likely to / and break the rewrite rule, basically rewriting everything to root. This is the wrong behaviour in 99% of the cases, but there is no warning since it's valid syntax.

So you need to be careful if using envsubst at any dollar sign in the YAML file which will be replaced as it looks like a variable.

@afirth
Copy link
Member

afirth commented Feb 1, 2021

@bedeabza here's an esoteric trick for you (bash) to envsubst only exported vars that start with _

I will add my two cents here, even though my issue was a "user problem", but in case anyone generates their YAML using envsubst they might encounter this which took me a while to find and fix.

When using nginx.ingress.kubernetes.io/rewrite-target: /$1 the last part $1 will get replaced by envsubst most likely to / and break the rewrite rule, basically rewriting everything to root. This is the wrong behaviour in 99% of the cases, but there is no warning since it's valid syntax.

So you need to be careful if using envsubst at any dollar sign in the YAML file which will be replaced as it looks like a variable.

$ cat in
my $_BAR
not $foo
$ export _BAR=bar
$ envsubst "$(printf '${%s} ' ${!_*})" < in > out
$ cat out
my bar
not $foo

Can't recall where I learnt the incantation but I hope it helps

@shydrate
Copy link

I am facing the same issue using Nginx Ingress on AWS. Unfortunately it took me 10+ hours to find this thread. :(

Could you share the ingress config, I am getting same error(404 error not found).
There are 2 frontend services running under same domain. Service1 is working but service2 is giving error. Thanks in advance. this line is already present nginx.ingress.kubernetes.io/rewrite-target: /. I am using aws eks alog with application load balancer with nginx.

          - path: /*
            backend:
              serviceName: service1
              servicePort: 80
          - path: /form/*
            backend:
              serviceName: service2
              servicePort: 8086

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests