net/http: go server behind nginx require read entire body before writing response #22209

Tevic · 2017-10-11T04:18:18Z

What version of Go are you using (`go version`)?

go version go1.9.1 windows/amd64

Does this issue reproduce with the latest release?

YES

What operating system and processor architecture are you using (`go env`)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=E:\Documents\Develop\WorkSpace\Go
set GORACE=
set GOROOT=E:\Documents\Develop\RunTime\Go\x64
set GOTOOLDIR=E:\Documents\Develop\RunTime\Go\x64\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=D:\CommonSoft\MSYS2\tmp\go-build716063943=/tmp/go-build -gno-record-gcc-switches
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

What did you do?

Start Go Server

package main

import (
	"log"
	"net/http"
)

func main() {
	MockServer()
}

func MockServer() {
	http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		w.WriteHeader(400)
		w.Write([]byte("HELLO"))
	})
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Start nginx
nginx.conf

upstream nodes {
        server 127.0.0.1:8080 max_fails=0;
}
server {
        listen 80;
        server_name ~.*;

        access_log logs/access.log main;
        error_log logs/error.log;

        client_max_body_size 1024m;
        client_body_buffer_size 512K;
        proxy_next_upstream error timeout non_idempotent;
        client_body_temp_path   client_body_temp_path 3 2;
        location / {
                proxy_http_version 1.1;
                proxy_pass http://nodes;
                break;
        }
}

Make A POST Request

package main

import (
	"bytes"
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
)

func main() {
	resp, err := http.Post("http://127.0.0.1:80", "application/octet-stream", bytes.NewReader(make([]byte, 1024*1024)))
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	respBuf, err := httputil.DumpResponse(resp, true)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(respBuf))
}

What did you expect to see?

Response:

HTTP/1.1 400 Bad Request
Content-Length: 5
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Wed, 11 Oct 2017 04:16:07 GMT
Server: nginx/1.13.0

HELLO

What did you see instead?

Resp:

HTTP/1.1 502 Bad Gateway
Content-Length: 173
Connection: keep-alive
Content-Type: text/html
Date: Wed, 11 Oct 2017 04:02:37 GMT
Server: nginx/1.13.0

<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.13.0</center>
</body>

Nginx Error:

2017/10/11 12:02:37 [error] 11956#2392: *49 WSASend() failed (10054: An existing connection was forcibly closed by the remote host) while sending request to upstream, client: 127.0.0.1, server: ~.*, request: "POST / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "127.0.0.1:80"

If i send request without body or the server side read all the body, response will be as expected.
It seems golang should ensure that a FIN packet is sent before any RST packet.
https://trac.nginx.org/nginx/ticket/1037

The text was updated successfully, but these errors were encountered:

nvartolomei · 2017-10-11T09:23:34Z

It seems golang should ensure that a FIN packet is sent before any RST packet.

Looks like server already tries to do this

go/src/net/http/server.go

Lines 1611 to 1623 in 6013052

    
           // closeWrite flushes any outstanding data and sends a FIN packet (if 
        
           // client is connected via TCP), signalling that we're done. We then 
        
           // pause for a bit, hoping the client processes it before any 
        
           // subsequent RST. 
        
           // 
        
           // See https://golang.org/issue/3595 
        
           func (c *conn) closeWriteAndWait() { 
        
           	c.finalFlush() 
        
           	if tcp, ok := c.rwc.(closeWriter); ok { 
        
           		tcp.CloseWrite() 
        
           	} 
        
           	time.Sleep(rstAvoidanceDelay) 
        
           }

riiiiizzzzzohmmmmm · 2017-10-19T06:52:47Z

I too am experiencing this issue running a Go server behind nginx. Any time my server receives a large enough POST payload (8K?) that fails Authorization via headers, I felt I should respond with http.StatusUnauthorized without reading the entire body.

I've read various relevant issue threads, including https://golang.org/issue/3595, but haven't found a resolution other than reading and discarding the request body, but that seems like an exploitable situation to me.

huguesb · 2017-10-19T16:24:16Z

Have you tried disabling proxy request buffering in your nginx config?
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

riiiiizzzzzohmmmmm · 2017-10-23T22:00:59Z

@huguesb that is a really helpful tip. i'm going to give it a try and report back. thanks!

Tevic · 2017-10-24T02:17:27Z

I add the config below in nginx config and it works, i am confused why this issue related to tcp keepalive.

#add to server block
proxy_http_version 1.1;
proxy_set_header Connection "";
 
#add to upstream block
keepalive 120;

tombergan · 2017-11-03T21:20:17Z

If someone can distill this to a pure-Go test, we can take a closer look, but as of now there's no clear bug for us to fix. As @nvartolomei mentioned earlier, we already try to CloseWrite the connection to prevent sending an early RST. Perhaps we're not doing that in the right way, or not in all cases, but at the moment I can't tell if the problem is in Go or nginx.

bradfitz · 2017-11-07T21:00:43Z

I think this is due to this part of func (*chunkWriter) writeHeader:

        // Per RFC 2616, we should consume the request body before
        // replying, if the handler hasn't already done so. But we
        // don't want to do an unbounded amount of reading here for
        // DoS reasons, so we only try up to a threshold.
        // TODO(bradfitz): where does RFC 2616 say that? See Issue 15527
        // about HTTP/1.x Handlers concurrently reading and writing, like
        // HTTP/2 handlers can do. Maybe this code should be relaxed?

And it references #15527.

The threshold it uses is 256k:

// maxPostHandlerReadBytes is the max number of Request.Body bytes not
// consumed by a handler that the server will read from the client
// in order to keep a connection alive. If there are more bytes than
// this then the server to be paranoid instead sends a "Connection:
// close" response.
//
// This number is approximately what a typical machine's TCP buffer
// size is anyway.  (if we have the bytes on the machine, we might as
// well read them)
const maxPostHandlerReadBytes = 256 << 10

But it's not clear that changing this behavior would make nginx happy anyway.

I think we have enough information in this bug to repro (with nginx), though, and then make a decision whether we need to make changes.

I'll flag this as HelpWanted and NeedsInvestigation. Maybe somebody can play around.

smasher164 · 2017-11-20T17:42:40Z

Trying to reproduce this example in pure go led to the correct behavior.
client.go

package main

import (
	"bytes"
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
)

func main() {
	resp, err := http.Post("http://127.0.0.1:80", "application/octet-stream", bytes.NewReader(make([]byte, 1024*1024)))
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	respBuf, err := httputil.DumpResponse(resp, true)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(respBuf))
}

backend.go

package main

import (
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		w.WriteHeader(400)
		w.Write([]byte("HELLO"))
	})
	log.Fatal(http.ListenAndServe(":8080", nil))
}

proxy.go

package main

import (
	"log"
	"net/http"
	"net/http/httputil"
	"net/url"
)

func main() {
	u, err := url.Parse("http://127.0.0.1:8080")
	if err != nil {
		log.Fatalln(err)
	}
	proxy := httputil.NewSingleHostReverseProxy(u)
	http.Handle("/", proxy)
	log.Fatal(http.ListenAndServe(":80", nil))
}

Output

HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 5
Content-Type: text/plain; charset=utf-8
Date: Mon, 20 Nov 2017 17:41:41 GMT

HELLO

marcelmiguel · 2018-02-19T17:07:54Z

I also have an error similar to this one.
There is a nginx in front of a golang API.
The first request works, but the second fails.
The solution of Tevic does not work for me.
Calling the API fails from Chrome, from postman works ok.

herrberk · 2018-03-19T16:18:40Z

I am experiencing a similar issue as well. The following code reproduces the issue in pure-Go: @tombergan
https://play.golang.org/p/G8F0X4DQCDn

For HTTP 1.1 with Transfer-Encoding: chunked, with Content-Length: -1 the request body is closed if we attempt to write part of the response immediately after receiving the first chunk, which is a big problem if you are doing some kind of streaming operation.

@bradfitz When I look at the source code you mentioned above, a possible solution is to change the if condition to check for Content-Length > 0 instead of Content-Length != 0:

go/src/net/http/server.go

Line 1282 in 5a4e098

if w.req.ContentLength != 0 && !w.closeAfterReply {

This minor change fixes the issue for the code I posted, at least for my use case. Please take a look at it, since this is a very big issue for us affecting many users.

nextgentm1 · 2018-03-28T19:19:50Z

I am having the same annoying problem. The workaround @herrberk posted seems to fix the issue locally. @bradfitz, @tombergan is it possible to add this fix to the repo?

ghost · 2018-04-11T05:41:24Z

I had a similar issue while running go web server behind nginx proxy.
Nginx was returning error HTTP/1.1 502 Bad Gateway but net/http server without nginx was working fine.

Error was resolved after restarting nginx.

herrberk · 2018-04-11T12:01:54Z

@DineshBhosale were you using Transfer-Encoding:chunked ? Asking because the example I gave above, fails to receive all the chunks, and I am not using any reverse proxy at all.

ghost · 2018-04-11T22:15:23Z

This is the response header for some static files that are being served by
http.ServeFile(w, r, file2serve)

Accept-Ranges: bytes
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/javascript
Date: Thu, 12 Apr 2018 03:29:39 GMT
Last-Modified: Wed, 11 Apr 2018 05:32:15 GMT
Server: nginx/1.12.2
Transfer-Encoding: chunked
Vary: Accept-Encoding

Request header

GET /abc/main.dart.bootstrap.js HTTP/1.1
Host: www.example.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Accept: */*
DNT: 1
Referer: https:/www.example.com/abc/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: username=; sid=3; uid=1; ses=nnQvWdtPqmi32Q32Ne52

It works fine almost all the time but once it fails it is never able to recover and serve any request and I had to restart nginx to make it work again.

Also I noticed that http.ServeFile(w, r, file2serve) is adding Transfer-Encoding: chunked for files that are big and have Content-Length > 2000

herrberk · 2018-04-11T23:26:34Z

@DineshBhosale You are sending a GET request, so in your case the request is completely read before the server responds to your request. What we are talking about here is when sending a POST request, if the client request body is large, and server starts responding before completing reading the entire request body, the connection closes and data loss occurs. This issue occurs regardless of the go server being behind a reverse proxy or not. Here is my example to reproduce the issue:
https://play.golang.org/p/G8F0X4DQCDn

ghost · 2018-04-12T02:25:54Z

Ok thanks for the info. I don't know why I get error 503 for https traffic , it happened again on my end. Sometimes works sometimes don't. Happens for get requests as well.

marcelmiguel · 2018-05-17T07:17:17Z

I previously posted I had a similar error.
But my error was due to a panic error, that caused NGINX to show the 502 error.
So right now i'm not affected by this error.

hcconquer · 2018-05-31T14:00:37Z

@huguesb

Have you tried disabling proxy request buffering in your nginx config?
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

it seems not works, did you use tcpdump to verify ?

wskinner · 2018-06-22T23:26:27Z

I'm having a similar problem. I have a go server behind an nginx reverse proxy via nginx-ingress in kubernetes. My POST request body is empty, and restarting nginx didn't solve it. If I query my server without going through nginx, it works.

mxcjohn · 2022-05-26T08:24:10Z

hi @riiiiizzzzzohmmmmm I face this problem recently. there is a nginx proxy before my server ,and I foud that the client side can get the server response but the nginx will record a "connection reset by peer" error.
now I have to read the post data even though I don't care about it in able to solve the nginx error for temporary. It is not safe for my server .
I am confusing that It is only occur when the post body size bigger than some value. Do you know the reason ? Or which param decide the value ? And do you have better way to solve this situation ? I'm watting for your reply ,thank you so much.

ZekeLu · 2022-05-27T12:15:42Z

I think the behavior described in #22209 (comment) is reasonable. The solution is to configure nginx such that it only passes big requests to the upstream on the path that accepts big requests (means that it will read the whole request body), and rejects all other big requests.

client_max_body_size can be applied to Location, so we can configure nginx like this:

 upstream nodes {
         server 127.0.0.1:8080 max_fails=0;
 }
 server {
         listen 80;
         server_name ~.*;
 
         access_log logs/access.log main;
         error_log logs/error.log;
 
-        client_max_body_size 1024m;
+        client_max_body_size 256K;
         client_body_buffer_size 512K;
         proxy_next_upstream error timeout non_idempotent;
         client_body_temp_path   client_body_temp_path 3 2;
         location / {
                 proxy_http_version 1.1;
                 proxy_pass http://nodes;
                 break;
         }
+        # the upstream is expected to accept big request on this path
+        location /paths/accepts/big/request {
+                client_max_body_size 1024m;
+                proxy_http_version 1.1;
+                proxy_pass http://nodes;
+                break;
+        }
 }

Tevic changed the title ~~net/http: go server behind nginx should read entire body before writing response~~ net/http: go server behind nginx require read entire body before writing response Oct 11, 2017

tombergan added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 3, 2017

bradfitz added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Nov 7, 2017

bradfitz added this to the Go1.11 milestone Nov 7, 2017

bradfitz assigned bradfitz and tombergan Nov 7, 2017

bradfitz added help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 7, 2017

bradfitz unassigned bradfitz and tombergan Nov 7, 2017

gopherbot removed the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Nov 7, 2017

bradfitz modified the milestones: Go1.11, Unplanned Nov 7, 2017

herrberk mentioned this issue May 4, 2018

net/http: support concurrent Request.Body reads & ResponseWriter.Write calls in HTTP/1.x server #15527

Closed

mohammed90 mentioned this issue Jul 24, 2020

sometimes caddy closes connection to client prematurely caddyserver/caddy#3557

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: go server behind nginx require read entire body before writing response #22209

net/http: go server behind nginx require read entire body before writing response #22209

Tevic commented Oct 11, 2017 •

edited

Loading

nvartolomei commented Oct 11, 2017 •

edited

Loading

riiiiizzzzzohmmmmm commented Oct 19, 2017

huguesb commented Oct 19, 2017

riiiiizzzzzohmmmmm commented Oct 23, 2017

Tevic commented Oct 24, 2017 •

edited

Loading

tombergan commented Nov 3, 2017

bradfitz commented Nov 7, 2017

smasher164 commented Nov 20, 2017

marcelmiguel commented Feb 19, 2018 •

edited

Loading

herrberk commented Mar 19, 2018 •

edited

Loading

nextgentm1 commented Mar 28, 2018

ghost commented Apr 11, 2018

herrberk commented Apr 11, 2018

ghost commented Apr 11, 2018 •

edited by ghost

Loading

herrberk commented Apr 11, 2018

ghost commented Apr 12, 2018

marcelmiguel commented May 17, 2018

hcconquer commented May 31, 2018

wskinner commented Jun 22, 2018

mxcjohn commented May 26, 2022

ZekeLu commented May 27, 2022 •

edited

Loading

net/http: go server behind nginx require read entire body before writing response #22209

net/http: go server behind nginx require read entire body before writing response #22209

Comments

Tevic commented Oct 11, 2017 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

nvartolomei commented Oct 11, 2017 • edited Loading

riiiiizzzzzohmmmmm commented Oct 19, 2017

huguesb commented Oct 19, 2017

riiiiizzzzzohmmmmm commented Oct 23, 2017

Tevic commented Oct 24, 2017 • edited Loading

tombergan commented Nov 3, 2017

bradfitz commented Nov 7, 2017

smasher164 commented Nov 20, 2017

marcelmiguel commented Feb 19, 2018 • edited Loading

herrberk commented Mar 19, 2018 • edited Loading

nextgentm1 commented Mar 28, 2018

ghost commented Apr 11, 2018

herrberk commented Apr 11, 2018

ghost commented Apr 11, 2018 • edited by ghost Loading

herrberk commented Apr 11, 2018

ghost commented Apr 12, 2018

marcelmiguel commented May 17, 2018

hcconquer commented May 31, 2018

wskinner commented Jun 22, 2018

mxcjohn commented May 26, 2022

ZekeLu commented May 27, 2022 • edited Loading

Tevic commented Oct 11, 2017 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

nvartolomei commented Oct 11, 2017 •

edited

Loading

Tevic commented Oct 24, 2017 •

edited

Loading

marcelmiguel commented Feb 19, 2018 •

edited

Loading

herrberk commented Mar 19, 2018 •

edited

Loading

ghost commented Apr 11, 2018 •

edited by ghost

Loading

ZekeLu commented May 27, 2022 •

edited

Loading