Invalid handling of chunked requests: Transfer-encoding "chunked" changed automatically to content-length #4838

OscarNeira · 2019-12-17T10:20:39Z

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

NGINX Ingress controller version:
Release: 0.26.1
Build: git-2de5a893a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: openresty/1.15.8.2

Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: On-prem and Minikube
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

What happened:

I have a java client that sends chunked HTTP POST requests, and it works fine when it is connected directly to my service using the node IP and service port. The problem is when the client sends requests through the Ingress. Requests with chunked data lower than 2m are failling and one or two passing and it feels that is random. After checking the logs, I found out that that the nginx ingress controller is changing the header "Transfer-Encoding: chunked" to "Content-Length".

What you expected to happen:

Ingress controller should not buffer the incoming requests if proxy_request_buffering is off.

How to reproduce it (as minimally and precisely as possible):

Start Minikube with 1.13.
Enable Ingress
Configure ingress and turn off proxy_buffering and proxy_request_buffering. Here is the config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: spectrum-replicator-http-api-com
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
spec:
  rules:
    - host: spectrum-replicator-http-api.com
      http:
        paths:
          - path: /
            backend:
              serviceName: spectrum-replicator
              servicePort: 10010

Set Ingress controller in debug mode
Send POST HTTP request with chunked data. In my case, the java client sends the same request with different payload sizes 3k, 33k, 500k, 1m, 2m, 3m, 5m, 7m .....
Here is one of the failing requests. The ingress controller response with 200 OK and zero content. The Nginx ingress controller gets the chunk data from the client and changes the request to use Content-Length: 38. (including an example with logs from the same request with the correct headers)

http request line: "POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-2/payload HTTP/1.1"
  lua rewrite handler, uri:"/api/1/space/spectrum-aa-1/objects/test-1576243368918/load-test-2/payload" c:1
  2019/12/13 13:22:50 [debug] 47#47: *310 looking up Lua code cache with key '=rewrite_by_lua(nginx.conf:450)nhli_810dc96928b7314c9cd60e7b526c7375'
  2019/12/13 13:22:50 [debug] 47#47: *310 lua creating new thread
  2019/12/13 13:22:50 [debug] 47#47: *310 lua reset ctx
  2019/12/13 13:22:50 [debug] 47#47: *310 http cleanup add: 00005600766C5A38
  2019/12/13 13:22:50 [debug] 47#47: *310 lua run thread, top:0 c:1
  2019/12/13 13:22:50 [debug] 47#47: *310 add cleanup: 00005600766C5A90
  2019/12/13 13:22:50 [debug] 47#47: *310 lua resume returned 0
  2019/12/13 13:22:50 [debug] 47#47: *310 lua light thread ended normally
  2019/12/13 13:22:50 [debug] 47#47: *310 lua deleting light thread
  2019/12/13 13:22:50 [debug] 47#47: *310 post rewrite phase: 7
  2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 8
  2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 9
  2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 10
  2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 11
  2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 12
  2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 13
  2019/12/13 13:22:50 [debug] 47#47: *310 access phase: 14
  2019/12/13 13:22:50 [debug] 47#47: *310 post access phase: 15
  2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 16
  2019/12/13 13:22:50 [debug] 47#47: *310 generic phase: 17
  2019/12/13 13:22:50 [debug] 47#47: *310 posix_memalign: 0000560076664310:4096 @16
  2019/12/13 13:22:50 [debug] 47#47: *310 http client request body preread 59
  2019/12/13 13:22:50 [debug] 47#47: *310 http request body chunked filter
  2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 0000560076661912, size: 59 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 63 s:0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:3
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 54 s:4
  2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 0000560076661921, size: 44 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:5
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:6
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 31 s:0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 34 s:1
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:3
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 36 s:4
  2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 000056007666193B, size: 18 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:5
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:6
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 36 s:0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:3
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 52 s:4
  2019/12/13 13:22:50 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766612E0, pos 0000560076661946, size: 7 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:5
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:6
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 30 s:0
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:1
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:8
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0D s:9
  2019/12/13 13:22:50 [debug] 47#47: *310 http chunked byte: 0A s:10
  2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:1 f:0 0000560076661915, pos 0000560076661915, size: 12 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:1 f:0 0000560076661927, pos 0000560076661927, size: 20 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:1 f:0 0000560076661940, pos 0000560076661940, size: 6 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http body new buf t:0 f:0 0000000000000000, pos 0000000000000000, size: 0 file: 0, size: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http init upstream, client timer: 0
  2019/12/13 13:22:50 [debug] 47#47: *310 http map started
  2019/12/13 13:22:50 [debug] 47#47: *310 http map: "" ""
  2019/12/13 13:22:50 [debug] 47#47: *310 http map started
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "7ec5a68f74b7e62717c59e19d8653836"
  2019/12/13 13:22:50 [debug] 47#47: *310 http map: "" "7ec5a68f74b7e62717c59e19d8653836"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "Host"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Request-ID"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "7ec5a68f74b7e62717c59e19d8653836"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Real-IP"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "192.168.64.1"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-For"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "192.168.64.1"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-Host"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-Port"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "80"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Forwarded-Proto"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "http"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "X-Scheme"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "http"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: "Content-Length"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script var: "38"
  2019/12/13 13:22:50 [debug] 47#47: *310 http script copy: ""
  "POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-2/payload HTTP/1.1
  Host: spectrum-replicator-http-api.com:80
  X-Request-ID: 7ec5a68f74b7e62717c59e19d8653836
  X-Real-IP: 192.168.64.1
  X-Forwarded-For: 192.168.64.1
  X-Forwarded-Host: spectrum-replicator-http-api.com:80
  X-Forwarded-Port: 80
  X-Forwarded-Proto: http
  X-Scheme: http
  Content-Length: 38
  Content-Type: application/octet-stream

Anything else we need to know:
In this particular test, the tcpdumps show that the client is sending 5 segments of data to the nginx and the ingress controller resend the chunked data in 2 segments. Here are the logs of the same request but the ingress controller does nothing to the buffers or headers.

http request line: "POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-1/payload HTTP/1.1"
  2019/12/13 13:22:49 [debug] 47#47: *310 http client request body preread 17
  2019/12/13 13:22:49 [debug] 47#47: *310 http request body chunked filter
  2019/12/13 13:22:49 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766C4090, pos 00005600766C46C2, size: 17 file: 0, size: 0
  2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 63 s:0
  2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0D s:1
  2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0A s:3
  2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 54 s:4
  2019/12/13 13:22:49 [debug] 47#47: *310 http body chunked buf t:1 f:0 00005600766C4090, pos 00005600766C46D1, size: 2 file: 0, size: 0
  2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0D s:5
  2019/12/13 13:22:49 [debug] 47#47: *310 http chunked byte: 0A s:6
  2019/12/13 13:22:49 [debug] 47#47: *310 http body new buf t:1 f:0 00005600766C46C5, pos 00005600766C46C5, size: 12 file: 0, size: 0
  2019/12/13 13:22:49 [debug] 47#47: *310 malloc: 0000560076663300:8192
  2019/12/13 13:22:49 [debug] 47#47: *310 http read client request body
  2019/12/13 13:22:49 [debug] 47#47: *310 recv: eof:0, avail:0
  2019/12/13 13:22:49 [debug] 47#47: *310 http client request body recv -2
  2019/12/13 13:22:49 [debug] 47#47: *310 http client request body rest 3
  2019/12/13 13:22:49 [debug] 47#47: *310 event timer add: 3: 60000:22794549
  2019/12/13 13:22:49 [debug] 47#47: *310 http init upstream, client timer: 1
  2019/12/13 13:22:49 [debug] 47#47: *310 event timer del: 3: 22794549
  2019/12/13 13:22:49 [debug] 47#47: *310 http map started
  2019/12/13 13:22:49 [debug] 47#47: *310 http map: "" ""
  2019/12/13 13:22:49 [debug] 47#47: *310 http map started
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "a9991ec03b9caa426e510b8ae40f2a13"
  2019/12/13 13:22:49 [debug] 47#47: *310 http map: "" "a9991ec03b9caa426e510b8ae40f2a13"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "Host"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Request-ID"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "a9991ec03b9caa426e510b8ae40f2a13"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Real-IP"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "192.168.64.1"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-For"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "192.168.64.1"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-Host"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "spectrum-replicator-http-api.com:80"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-Port"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "80"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Forwarded-Proto"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "http"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "X-Scheme"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "http"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: ""
  2019/12/13 13:22:49 [debug] 47#47: *310 http script copy: "Transfer-Encoding"
  2019/12/13 13:22:49 [debug] 47#47: *310 http script var: "chunked"
  "POST /api/1/space/spectrum-aa-1/objects/test-1576243368918%2Fload-test-1/payload HTTP/1.1
  Host: spectrum-replicator-http-api.com:80
  X-Request-ID: a9991ec03b9caa426e510b8ae40f2a13
  X-Real-IP: 192.168.64.1
  X-Forwarded-For: 192.168.64.1
  X-Forwarded-Host: spectrum-replicator-http-api.com:80
  X-Forwarded-Port: 80
  X-Forwarded-Proto: http
  X-Scheme: http
  Transfer-Encoding: chunked
  Content-Type: application/octet-stream

Here is the nginx config:

## start server spectrum-replicator-http-api.com
	server {
		server_name spectrum-replicator-http-api.com ;
		
		listen 80  ;
		listen 443  ssl http2 ;
		
		set $proxy_upstream_name "-";
		
		ssl_certificate_by_lua_block {
			certificate.call()
		}
		
		location / {
			
			set $namespace      "oneira-sandbox";
			set $ingress_name   "spectrum-replicator-http-api-com";
			set $service_name   "spectrum-replicator";
			set $service_port   "10010";
			set $location_path  "/";
			
			rewrite_by_lua_block {
				lua_ingress.rewrite({
					force_ssl_redirect = false,
					ssl_redirect = true,
					force_no_ssl_redirect = false,
					use_port_in_redirects = false,
				})
				balancer.rewrite()
				plugins.run()
			}
			
			header_filter_by_lua_block {
				
				plugins.run()
			}
			body_filter_by_lua_block {
				
			}
			
			log_by_lua_block {
				
				balancer.log()
				
				monitor.call()
				
				plugins.run()
			}
			
			port_in_redirect off;
			
			set $balancer_ewma_score -1;
			set $proxy_upstream_name "oneira-sandbox-spectrum-replicator-10010";
			set $proxy_host          $proxy_upstream_name;
			set $pass_access_scheme  $scheme;
			set $pass_server_port    $server_port;
			set $best_http_host      $http_host;
			set $pass_port           $pass_server_port;
			
			set $proxy_alternative_upstream_name "";
			
			client_max_body_size                    0;
			
			proxy_set_header Host                   $best_http_host;
			
			# Pass the extracted client certificate to the backend
			
			# Allow websocket connections
			proxy_set_header                        Upgrade           $http_upgrade;
			
			proxy_set_header                        Connection        $connection_upgrade;
			
			proxy_set_header X-Request-ID           $req_id;
			proxy_set_header X-Real-IP              $remote_addr;
			
			proxy_set_header X-Forwarded-For        $remote_addr;
			
			proxy_set_header X-Forwarded-Host       $best_http_host;
			proxy_set_header X-Forwarded-Port       $pass_port;
			proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
			
			proxy_set_header X-Scheme               $pass_access_scheme;
			
			# Pass the original X-Forwarded-For
			proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
			
			# mitigate HTTPoxy Vulnerability
			# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
			proxy_set_header Proxy                  "";
			
			# Custom headers to proxied server
			
			proxy_connect_timeout                   5s;
			proxy_send_timeout                      60s;
			proxy_read_timeout                      60s;
			
			proxy_buffering                         off;
			proxy_buffer_size                       128k;
			proxy_buffers                           4 128k;
			
			proxy_max_temp_file_size                0;
			
			proxy_request_buffering                 off;
			proxy_http_version                      1.1;
			
			proxy_cookie_domain                     off;
			proxy_cookie_path                       off;
			
			# In case of errors try the next upstream server before returning an error
			proxy_next_upstream                     error timeout;
			proxy_next_upstream_timeout             0;
			proxy_next_upstream_tries               3;
			
			proxy_pass http://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
	}
	## end server spectrum-replicator-http-api.com

The text was updated successfully, but these errors were encountered:

aledbf · 2019-12-30T17:19:48Z

@OscarNeira if you can provide a self-contained example that reproduces the issue, I can take a look.

OscarNeira · 2020-01-07T10:06:09Z

@OscarNeira if you can provide a self-contained example that reproduces the issue, I can take a look.

Yes, I will.

OscarNeira · 2020-02-06T11:53:53Z

Hi @aledbf , I managed to reproduce the issue in a way that you can also try. and added some steps:

Created new simple Java client here:

Get the example here: https://github.com/OscarNeira/simple-java-client

Use a mock server to inspect the incoming request from the Nginx.

You can use this version that includes the tcpdump layer in docker: https://github.com/OscarNeira/mockserver/tree/add/tcpdumps
Deployed that in my local cluster.
Use the Java client to create expectations: Run the test in the project (CreateSimpleExpectation and CreateSimpleExpectationPost)
Test expectations with curl -v GET http://<INGRESS_HOST>/delay/1

Use tcpdump in mockserver pod Kubectl exec –it <POD_NAME> -- /bin/bash

Once inside the pod run tcpdump -w tcpdump-ingress-chunk-1.pcap
New terminal copy the file to local folder Kubectl cp <NAMESPACE>/ <POD_NAME>:tcpdump-ingress-chunk-1.pcap ~/tcpdump-ingress-chunk-1.pcap

Inspect incoming requests.

I verified that the client is sending Transfer-Encoding: chunked using Wireshark
With the attached file tcpdump-ingress-chunk-1.pcap you can see that Nginx is changing the header to Content-Length

tcpdump-ingress-chunk-1.pcap.zip

Here is the ingress config:

kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
  creationTimestamp: "2020-02-04T10:01:28Z"
  generation: 1
  name: mock-server-chunk-config
  namespace: mockserver
  resourceVersion: "5550485"
  selfLink: /apis/extensions/v1beta1/namespaces/mockserver/ingresses/mock-server-chunk-config
  uid: 4fd93f88-4735-11ea-a94f-f8b1569bea58
spec:
  rules:
  - host: mockserver.jesus-microk8s.com
    http:
      paths:
      - backend:
          serviceName: mockserver
          servicePort: 1080
        path: /
status:
  loadBalancer:
    ingress:
    - ip: 127.0.0.1```

aledbf · 2020-02-06T12:08:08Z

@OscarNeira please update to 0.28.0 and test again.

aledbf · 2020-02-06T12:08:58Z

Today I will try to reproduce this issue. Thank you for the procedure to reproduce the issue.

OscarNeira · 2020-02-06T13:16:08Z

@OscarNeira please update to 0.28.0 and test again.

same results with 0.28.0

aledbf · 2020-02-08T21:25:51Z

@OscarNeira can you try to provide a simpler way to reproduce this?

Something like https://gist.github.com/aledbf/266940de7569a1163b9e1c085aa4e771

OscarNeira · 2020-02-10T06:54:43Z

yep, I will try.

OscarNeira · 2020-02-11T12:40:07Z

Hi @aledbf , Added more details to the repo that just created. Here is the link to the script and the yaml files are in the same folder. Let me know if you need anything else. https://github.com/OscarNeira/simple-java-client/blob/master/k8s/start.

aledbf · 2020-02-11T12:48:21Z

@OscarNeira

./k8s/start.sh 
192.168.39.183 example.mockserver.com
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-wvxpv Is mock server Running
mockserver-67c755fd7b-wvxpv Is Running
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
latest: Pulling from oscneira/java_client
Digest: sha256:a4195528758fdb08a6212c5732dd396e1daa646ead659fd5daa61f5f7178160f
Status: Image is up to date for oscneira/java_client:latest
docker.io/oscneira/java_client:latest
Setup Mock Server
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.mockserver.client.ClientException: error:

	<html><head><title>Object moved</title></head><body>
	<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com">here</a>.</h2>
	</body></html>


 while submitted expectation:

	{
	  "id" : "4c4cc29f-f14f-409f-a39f-15330dffcab9",
	  "httpRequest" : {
	    "method" : "GET",
	    "path" : "/delay/1"
	  },
	  "times" : {
	    "unlimited" : true
	  },
	  "timeToLive" : {
	    "unlimited" : true
	  },
	  "httpResponse" : {
	    "body" : "OK",
	    "delay" : {
	      "timeUnit" : "SECONDS",
	      "value" : 1
	    }
	  }
	}

	at org.mockserver.client.MockServerClient.sendExpectation(MockServerClient.java:808)
	at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:47)
	at com.rest.JavaClient.createDelayMockServer(JavaClient.java:120)
	at com.rest.JavaClient.main(JavaClient.java:37)

OscarNeira · 2020-02-11T13:06:46Z

could you try to run this curl -v -k GET "http://example.mockserver.com/delay/1"

aledbf · 2020-02-11T13:07:52Z

curl -v -k GET "http://example.mockserver.com/delay/1"
* Could not resolve host: GET
curl: (6) Could not resolve host: GET
*   Trying 192.168.39.183:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.183) port 80 (#1)
> GET /delay/1 HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.17.7
< Date: Tue, 11 Feb 2020 13:07:36 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 186
< Connection: keep-alive
< Cache-Control: private
< Location: https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com
< X-Powered-By: ASP.NET
< 
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com">here</a>.</h2>
</body></html>
* Connection #1 to host example.mockserver.com left intact

OscarNeira · 2020-02-11T13:09:51Z

could you check that 192.168.39.183 example.mockserver.com exists in the /etc/hosts

aledbf · 2020-02-11T13:11:10Z

tail -2 /etc/hosts 

192.168.39.183 example.mockserver.com

OscarNeira · 2020-02-11T13:13:00Z

ok, could you try to open new command line/powershell/terminal and try again curl -v -k GET "http://example.mockserver.com/delay/1"

aledbf · 2020-02-11T13:14:20Z

curl -v -k  "http://example.mockserver.com/delay/1"
*   Trying 192.168.39.183:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.183) port 80 (#0)
> GET /delay/1 HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Server: nginx/1.17.7
< Date: Tue, 11 Feb 2020 13:14:09 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 186
< Connection: keep-alive
< Cache-Control: private
< Location: https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com
< X-Powered-By: ASP.NET
< 
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com">here</a>.</h2>
</body></html>
* Connection #0 to host example.mockserver.com left intact

OscarNeira · 2020-02-11T13:20:44Z

Ok, for some reason cannot find the host. I will delete mi minikube and run it clean.
~ at ☸️ minikube took 1m 7s
✦2 ➜ minikube delete
🔥 Deleting "minikube" in hyperkit ...
💀 Removed all traces of the "minikube" cluster.

~ took 6s
✦2 ➜ minikube start --kubernetes-version v1.13.6 --vm-driver="hyperkit" --cpus 4 --memory 5120
😄 minikube v1.7.2 on Darwin 10.14.6
✨ Using the hyperkit driver based on user configuration
🔥 Creating hyperkit VM (CPUs=4, Memory=5120MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.13.6 on Docker 19.03.5 ...

aledbf · 2020-02-11T13:23:02Z

@OscarNeira not sure what you mean. This is the log of the mockserver pod

k logs mockserver-67c755fd7b-fl58q

java  -Dfile.encoding=UTF-8 -jar /opt/mockserver/mockserver-netty-jar-with-dependencies.jar  -serverPort 1080 -logLevel INFO &

2020-02-11 13:21:20  org.mockserver.log.MockServerEventLog  INFO  started on port: 1080 
2020-02-11 13:21:20  org.mockserver.cli.Main  INFO  logger level is INFO, change using:
 - 'ConfigurationProperties.logLevel(String level)' in Java code,
 - '-logLevel' command line argument,
 - 'mockserver.logLevel' JVM system property or,
 - 'mockserver.logLevel' property value in 'mockserver.properties' 
2020-02-11 13:21:42  org.mockserver.log.MockServerEventLog  INFO  received request:

	{
	  "method" : "GET",
	  "path" : "/delay/1",
	  "headers" : {
	    "Host" : [ "example.mockserver.com" ],
	    "X-Request-ID" : [ "bb1f276c36ed631d3cecb4f988738766" ],
	    "X-Real-IP" : [ "192.168.39.1" ],
	    "X-Forwarded-For" : [ "192.168.39.1" ],
	    "X-Forwarded-Host" : [ "example.mockserver.com" ],
	    "X-Forwarded-Port" : [ "80" ],
	    "X-Forwarded-Proto" : [ "http" ],
	    "X-Scheme" : [ "http" ],
	    "User-Agent" : [ "curl/7.65.3" ],
	    "Accept" : [ "*/*" ],
	    "content-length" : [ "0" ]
	  },
	  "keepAlive" : true,
	  "secure" : false
	}
 
2020-02-11 13:21:43  org.mockserver.log.MockServerEventLog  INFO  returning response:

	{
	  "statusCode" : 302,
	  "reasonPhrase" : "Found",
	  "headers" : {
	    "Cache-Control" : [ "private" ],
	    "Content-Type" : [ "text/html; charset=utf-8" ],
	    "Location" : [ "https://www.hugedomains.com/domain_profile.cfm?d=mockserver&e=com" ],
	    "Server" : [ "Microsoft-IIS/8.5" ],
	    "X-Powered-By" : [ "ASP.NET" ],
	    "Date" : [ "Tue, 11 Feb 2020 13:21:40 GMT" ],
	    "Content-Length" : [ "186" ]
	  },
	  "body" : "<html><head><title>Object moved</title></head><body>\r\n<h2>Object moved to <a href=\"https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com\">here</a>.</h2>\r\n</body></html>\r\n"
	}

 for forwarded request

 in json:

	{
	  "method" : "GET",
	  "path" : "/delay/1",
	  "headers" : {
	    "Host" : [ "example.mockserver.com" ],
	    "X-Request-ID" : [ "bb1f276c36ed631d3cecb4f988738766" ],
	    "X-Real-IP" : [ "192.168.39.1" ],
	    "X-Forwarded-For" : [ "192.168.39.1" ],
	    "X-Forwarded-Host" : [ "example.mockserver.com" ],
	    "X-Forwarded-Port" : [ "80" ],
	    "X-Forwarded-Proto" : [ "http" ],
	    "X-Scheme" : [ "http" ],
	    "User-Agent" : [ "curl/7.65.3" ],
	    "Accept" : [ "*/*" ],
	    "content-length" : [ "0" ]
	  },
	  "keepAlive" : true,
	  "secure" : false
	}

 

 in curl:

	curl -v 'http://example.mockserver.com/delay/1' -H 'Host: example.mockserver.com' -H 'X-Request-ID: bb1f276c36ed631d3cecb4f988738766' -H 'X-Real-IP: 192.168.39.1' -H 'X-Forwarded-For: 192.168.39.1' -H 'X-Forwarded-Host: example.mockserver.com' -H 'X-Forwarded-Port: 80' -H 'X-Forwarded-Proto: http' -H 'X-Scheme: http' -H 'User-Agent: curl/7.65.3' -H 'Accept: */*' -H 'content-length: 0'

OscarNeira · 2020-02-11T13:33:35Z

could you try to run the image again docker run -it --rm=true oscneira/java_client

OscarNeira · 2020-02-11T13:41:47Z

Deleted all the objects

kubectl delete deploy mockserver
#kubectl delete ing mock-server-chunk-config-test
#kubectl delete svc mockserver

and run the script again

 ./k8s/start.sh 
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver created
deployment.apps/mockserver created
ingress.extensions/mock-server-chunk-config-test created
mockserver-75c9d49dc8-qp2kn Is mock server Running
mockserver-75c9d49dc8-qp2kn Is Running
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
latest: Pulling from oscneira/java_client
Digest: sha256:a4195528758fdb08a6212c5732dd396e1daa646ead659fd5daa61f5f7178160f
Status: Image is up to date for oscneira/java_client:latest
docker.io/oscneira/java_client:latest
Setup Mock Server
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Mock Server Ready
START http call Small Encoded file
InputStreamEntity DATA getContent: java.io.FileInputStream@3624da92
InputStreamEntity DATA getContentType: Content-Type: application/octet-stream
InputStreamEntity DATA getContentEncoding: null
InputStreamEntity DATA isStreaming: false
Executing request: POST http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-37318%2Ftest_createSingleSmallCall%2Fload-test-1/payload HTTP/1.1
Executing request Hearder Name x-spectrum-access-token
Executing request Hearder Value 
Executing request Hearder Name Content-Type
Executing request Hearder Value application/octet-stream
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-name
Executing request Hearder Value ["test_createSingleSmallCall"]
Executing request Hearder Name x-spectrum-meta-fs2_metadata_dataDefinitionId
Executing request Hearder Value Payload
Executing request Hearder Name x-spectrum-meta-wasZippedFirst
Executing request Hearder Value false
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdBy
Executing request Hearder Value FS2
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-header
Executing request Hearder Value ["test-header-value"]
Executing request Hearder Name source
Executing request Hearder Value G2_SB_DEV_INT
Executing request Hearder Name ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdOn
Executing request Hearder Value Tue Feb 11 13:38:58 GMT 2020
Executing request Hearder Name timestamp
Executing request Hearder Value 1581428338896
Executing request Hearder Name Connection
Executing request Hearder Value Keep-Alive
Executing request Hearder Name Accept-Encoding
Executing request Hearder Value gzip,deflate
----------------------------------------
HTTP/1.1 200 OK
OK
END http call Small Encoded file

OscarNeira · 2020-02-11T13:42:20Z

Could you try to delete and pull the latest image?

aledbf · 2020-02-11T13:56:21Z

Same result. I started a fresh minikube cluster

./k8s/start.sh 
Error from server (AlreadyExists): namespaces "mockserver" already exists
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-d8ncj Is mock server Running
Error from server (NotFound): pods "mockserver-67c755fd7b-d8ncj" not found
mockserver-67c755fd7b-d8ncj Is Running
Error from server (NotFound): pods "mockserver-67c755fd7b-d8ncj" not found
latest: Pulling from oscneira/java_client
Digest: sha256:a4195528758fdb08a6212c5732dd396e1daa646ead659fd5daa61f5f7178160f
Status: Image is up to date for oscneira/java_client:latest
docker.io/oscneira/java_client:latest
Setup Mock Server
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.mockserver.client.ClientException: error:

	<html><head><title>Object moved</title></head><body>
	<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com">here</a>.</h2>
	</body></html>


 while submitted expectation:

	{
	  "id" : "12d61441-3080-4e30-8cdc-98f7be25a51a",
	  "httpRequest" : {
	    "method" : "GET",
	    "path" : "/delay/1"
	  },
	  "times" : {
	    "unlimited" : true
	  },
	  "timeToLive" : {
	    "unlimited" : true
	  },
	  "httpResponse" : {
	    "body" : "OK",
	    "delay" : {
	      "timeUnit" : "SECONDS",
	      "value" : 1
	    }
	  }
	}

	at org.mockserver.client.MockServerClient.sendExpectation(MockServerClient.java:808)
	at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:47)
	at com.rest.JavaClient.createDelayMockServer(JavaClient.java:120)
	at com.rest.JavaClient.main(JavaClient.java:37)
Error from server (NotFound): pods "mockserver-67c755fd7b-d8ncj" not found

OscarNeira · 2020-02-12T07:18:41Z

Updated the project a bit to get more info, I feel that we are using an old image, created a new tag for this test. I also updated the script a bit. https://github.com/OscarNeira/simple-java-client/blob/master/k8s/start.sh

Could you check the /etc/hosts file has the correct minikube ip you can run minikube ip to verify that.
Update the repo to the latest master branch
and run the script again

aledbf · 2020-02-12T11:43:11Z

Same

 k8s/start.sh 
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-hclwj Is mock server Running
mockserver-67c755fd7b-hclwj Is Running
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
2.0.0: Pulling from oscneira/java_client
Digest: sha256:339cfa1271b078a816122e0137719e8af3f104b8d02c632fa5a85aa774ed8728
Status: Image is up to date for oscneira/java_client:2.0.0
docker.io/oscneira/java_client:2.0.0
Setup Mock Server
Create Expectation Delay 1 Sec
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.mockserver.client.ClientException: error:

	<html><head><title>Object moved</title></head><body>
	<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com">here</a>.</h2>
	</body></html>


 while submitted expectation:

	{
	  "id" : "9eac0e29-0dd1-4864-ac9f-964593925ec0",
	  "httpRequest" : {
	    "method" : "GET",
	    "path" : "/delay/1"
	  },
	  "times" : {
	    "unlimited" : true
	  },
	  "timeToLive" : {
	    "unlimited" : true
	  },
	  "httpResponse" : {
	    "body" : "OK",
	    "delay" : {
	      "timeUnit" : "SECONDS",
	      "value" : 1
	    }
	  }
	}

	at org.mockserver.client.MockServerClient.sendExpectation(MockServerClient.java:808)
	at org.mockserver.client.ForwardChainExpectation.respond(ForwardChainExpectation.java:47)
	at com.rest.JavaClient.createDelayMockServer(JavaClient.java:120)
	at com.rest.JavaClient.main(JavaClient.java:36)
 aledbf@me  ~/.../github/simple-java-client   master ●  curl example.mockserver.com/delay/1
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com">here</a>.</h2>
</body></html>

Also, reading the ingress controller pod logs, I see the 302 is returned by the app
192.168.39.1 - - [12/Feb/2020:11:41:07 +0000] "GET /delay/1 HTTP/1.1" 302 186 "-" "curl/7.65.3" 93 0.811 [mockserver-mockserver-1080] [] 172.17.0.4:1080 186 0.811 302 8af8c6049e5465827f23584f3c1edc0a

OscarNeira · 2020-02-12T12:37:35Z

Ok, it seems that the java client cannot create expectation rules. I will test using the minikube ip and let you know.

OscarNeira · 2020-02-12T12:41:39Z

It works as well with minikube IP could you share your minikube Ip? and I will create a new image tag with that change. After I push the change could you change the script start.sh and update the version?

docker pull oscneira/java_client:4.0.0
docker run -it --rm=true oscneira/java_client:4.0.0

OscarNeira · 2020-02-12T13:11:07Z

Here plan B: created expectations using curl. You can update the project to the latest version and run this one. https://github.com/OscarNeira/simple-java-client/blob/master/k8s/start-curl.sh

aledbf · 2020-02-12T13:20:39Z

It works as well with minikube IP could you share your minikube Ip?

Maybe you should read some environment variable? Hardcoding an IP is not very flexible

aledbf · 2020-02-12T13:21:00Z

Here plan B: created expectations using curl.

Same

 k8s/start-curl.sh 
Error from server (AlreadyExists): namespaces "mockserver" already exists
Context "minikube" modified.
service/mockserver unchanged
deployment.apps/mockserver unchanged
ingress.extensions/mock-server-chunk-config-test unchanged
mockserver-67c755fd7b-sqbx2 Is mock server Running
mockserver-67c755fd7b-sqbx2 Is Running
*   Trying 192.168.39.162:80...
* TCP_NODELAY set
* Connected to example.mockserver.com (192.168.39.162) port 80 (#0)
> GET /delay/1 HTTP/1.1
> Host: example.mockserver.com
> User-Agent: curl/7.65.3
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.17.7
< Date: Wed, 12 Feb 2020 13:19:03 GMT
< Content-Length: 16
< Connection: keep-alive
< 
* Connection #0 to host example.mockserver.com left intact
Delayed 1 secondtcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
5.0.0: Pulling from oscneira/java_client
Digest: sha256:be518c4eaaaf044352c413d9bf2b1ad82ae4568741f8a8de9f335c188a65fb7d
Status: Image is up to date for oscneira/java_client:5.0.0
docker.io/oscneira/java_client:5.0.0
START http call Small Encoded file
mockServerIngressEndpoint::http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-39473%2Ftest_createSingleSmallCall%2Fload-test-1/payload
InputStreamEntity DATA getContent: java.io.FileInputStream@1e7c7811
InputStreamEntity DATA getContentType: Content-Type: application/octet-stream
InputStreamEntity DATA getContentEncoding: null
InputStreamEntity DATA isStreaming: false
Executing request: POST http://example.mockserver.com/api/1/space/spectrum-aa-1/objects/jesus-test-39473%2Ftest_createSingleSmallCall%2Fload-test-1/payload HTTP/1.1
Executing request Hearder Name x-spectrum-access-token
Executing request Hearder Value 
Executing request Hearder Name Content-Type
Executing request Hearder Value application/octet-stream
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-name
Executing request Hearder Value ["test_createSingleSmallCall"]
Executing request Hearder Name x-spectrum-meta-fs2_metadata_dataDefinitionId
Executing request Hearder Value Payload
Executing request Hearder Name x-spectrum-meta-wasZippedFirst
Executing request Hearder Value false
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdBy
Executing request Hearder Value FS2
Executing request Hearder Name x-spectrum-meta-fs2_headers_test-header
Executing request Hearder Value ["test-header-value"]
Executing request Hearder Name source
Executing request Hearder Value G2_SB_DEV_INT
Executing request Hearder Name ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_ttl
Executing request Hearder Value 50000
Executing request Hearder Name x-spectrum-meta-fs2_metadata_createdOn
Executing request Hearder Value Wed Feb 12 13:19:09 GMT 2020
Executing request Hearder Name timestamp
Executing request Hearder Value 1581513549653
Executing request Hearder Name Connection
Executing request Hearder Value Keep-Alive
Executing request Hearder Name Accept-Encoding
Executing request Hearder Value gzip,deflate
----------------------------------------
HTTP/1.1 302 Found
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www.hugedomains.com/domain_profile.cfm?d=mockserver&amp;e=com">here</a>.</h2>
</body></html>

END http call Small Encoded file

OscarNeira · 2020-02-12T18:17:27Z

Did you get errors with this calls?

curl -X PUT "http://example.mockserver.com/expectation" -H "accept: */*" -H "Content-Type: application/json" -d '[{"httpRequest":{"path":"/api/1/space/.*","method":"POST"},"httpResponse":{"delay":{"timeUnit":"SECONDS","value":1},"statusCode":200,"reasonPhrase":"OK"}}]'

curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://example.mockserver.com/api/1/space/

OscarNeira · 2020-03-03T13:52:00Z

@aledbf Yep, it already had these

metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"

aledbf · 2020-03-03T13:53:17Z

@OscarNeira ok, then remove all the annotations, run the tests and post the log, please.

OscarNeira · 2020-03-03T13:53:57Z

@OscarNeira ok, then remove all the annotations, run the tests and post the log, please.

ok

OscarNeira · 2020-03-03T14:08:01Z

@aledbf same results:
nginx-controller-result-underscore3.log

OscarNeira · 2020-03-12T06:29:40Z

@aledbf any luck?

fejta-bot · 2020-06-10T07:13:48Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

OscarNeira · 2020-06-10T11:10:38Z

This is still a problem

fejta-bot · 2020-09-08T11:21:33Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2020-10-08T12:03:47Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2020-11-07T12:46:13Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2020-11-07T12:46:26Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

OscarNeira · 2020-11-09T07:26:26Z

never fix/ still and issue

herman-d · 2021-05-07T14:39:28Z

Hi @OscarNeira , do you manage to workaround the issue? I think I'm having the same issue where the transfer-encoding is replaced by content-length where it causes incorrect behaviour in the client.

OscarNeira · 2021-06-10T12:26:12Z

Hi @OscarNeira , do you manage to workaround the issue? I think I'm having the same issue where the transfer-encoding is replaced by content-length where it causes incorrect behaviour in the client.

@herman-d no, we still have same issue, we disable the chunck transfer for now.

owaisrehman · 2022-09-02T06:01:05Z

Hi Everyone,
Any updates on this behavior?
I'm more interested in why it is happening?
Recently I faced this issue as well
Environment Configuration:
base nginx version: nginx/1.19.9
ingress-nginx controller version: v1.1.1

xlccccc · 2024-04-21T03:45:39Z

any solution now?

longwuyuan · 2024-04-21T04:50:42Z

This issue was reported in Dec 2019. so is this the right place to track the problem

xlccccc · 2024-04-21T09:17:58Z

This issue was reported in Dec 2019. so is this the right place to track the problem

Do you have any solution? I don't want nginx to change the block encoding when doing reverse proxy.

jasonhwang-max · 2024-07-02T00:37:17Z

Has the issue been fixed in the latest version of nginx ingress controller?

longwuyuan · 2024-07-02T02:03:54Z

chunking is on by default in the version of Nginx being used in the controller. so if there is a problem, then its in the app code and not in the controller.

jasonhwang-max · 2024-07-02T04:32:58Z

Hi, @longwuyuan,
I know chunking is turned on by default.
Do you think so because there were no problems after testing the case in question?
So, do you think that transfer-encoding will never be changed to content-length in the controller, regardless of the environment?
A similar problem occurred recently and I would like to know the last confirmed details.

longwuyuan · 2024-07-02T07:21:00Z

@jasonhwang-max its not easy to generalize so

the issue description by the creator of this issue needs to see that the problem described is stale due to lots of updates in code and changes to features funcionalities
Please create your own issue with the test data of problem containing all the small tiny delicate details, that a reader needs
Enabling chunking is done by upstream nginx since nginx v1.24. Another user set chunking header in application code and ingress-nginx controller logged error for duplicate header because that header can be set only once per connection
Last time I tested in v1.10.1 of the controller, I did not see

changing the header "Transfer-Encoding: chunked" to "Content-Length".

jasonhwang-max · 2024-07-02T07:50:56Z

Thanks, @longwuyuan
Have an awesome day!

k8s-ci-robot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 10, 2020

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 8, 2020

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 8, 2020

k8s-ci-robot closed this as completed Nov 7, 2020

longwuyuan mentioned this issue Mar 26, 2024

since controller 1.10.0 (chart 4.10.0): ingress rejects duplicate "Transfer-Encoding: chunked" and returns 502 #11162

Closed

Invalid handling of chunked requests: Transfer-encoding "chunked" changed automatically to content-length #4838

Invalid handling of chunked requests: Transfer-encoding "chunked" changed automatically to content-length #4838

Comments

OscarNeira commented Dec 17, 2019 • edited Loading

aledbf commented Dec 30, 2019

OscarNeira commented Jan 7, 2020

OscarNeira commented Feb 6, 2020 • edited Loading

aledbf commented Feb 6, 2020

aledbf commented Feb 6, 2020

OscarNeira commented Feb 6, 2020

aledbf commented Feb 8, 2020

OscarNeira commented Feb 10, 2020

OscarNeira commented Feb 11, 2020

aledbf commented Feb 11, 2020

OscarNeira commented Feb 11, 2020

aledbf commented Feb 11, 2020

OscarNeira commented Feb 11, 2020

aledbf commented Feb 11, 2020

OscarNeira commented Feb 11, 2020

aledbf commented Feb 11, 2020

OscarNeira commented Feb 11, 2020 • edited Loading

aledbf commented Feb 11, 2020

OscarNeira commented Feb 11, 2020

OscarNeira commented Feb 11, 2020

OscarNeira commented Feb 11, 2020

aledbf commented Feb 11, 2020

OscarNeira commented Feb 12, 2020

aledbf commented Feb 12, 2020

OscarNeira commented Feb 12, 2020

OscarNeira commented Feb 12, 2020

OscarNeira commented Feb 12, 2020

aledbf commented Feb 12, 2020

aledbf commented Feb 12, 2020

OscarNeira commented Feb 12, 2020

OscarNeira commented Mar 3, 2020 • edited Loading

aledbf commented Mar 3, 2020

OscarNeira commented Mar 3, 2020

OscarNeira commented Mar 3, 2020 • edited Loading

OscarNeira commented Mar 12, 2020

fejta-bot commented Jun 10, 2020

OscarNeira commented Jun 10, 2020

fejta-bot commented Sep 8, 2020

fejta-bot commented Oct 8, 2020

fejta-bot commented Nov 7, 2020

k8s-ci-robot commented Nov 7, 2020

OscarNeira commented Nov 9, 2020 • edited Loading

herman-d commented May 7, 2021

OscarNeira commented Jun 10, 2021

owaisrehman commented Sep 2, 2022

xlccccc commented Apr 21, 2024

longwuyuan commented Apr 21, 2024

xlccccc commented Apr 21, 2024

jasonhwang-max commented Jul 2, 2024

longwuyuan commented Jul 2, 2024

jasonhwang-max commented Jul 2, 2024 • edited Loading

longwuyuan commented Jul 2, 2024

jasonhwang-max commented Jul 2, 2024

OscarNeira commented Dec 17, 2019 •

edited

Loading

OscarNeira commented Feb 6, 2020 •

edited

Loading

OscarNeira commented Feb 11, 2020 •

edited

Loading

OscarNeira commented Mar 3, 2020 •

edited

Loading

OscarNeira commented Mar 3, 2020 •

edited

Loading

OscarNeira commented Nov 9, 2020 •

edited

Loading

jasonhwang-max commented Jul 2, 2024 •

edited

Loading