Expect 100-Continue Panic

[murphymj25]

We have recently started to see the following panic intermittently across our fabio instance:

May 15 10:21:05 <hostname> fabio: panic: runtime error: invalid memory address or nil pointer dereference
May 15 10:21:05 <hostname> fabio: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x5463cf]
May 15 10:21:05 <hostname> fabio: goroutine 6219360 [running]:
May 15 10:21:05 <hostname> fabio: bufio.(*Writer).Available(...)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/bufio/bufio.go:607
May 15 10:21:05 <hostname> fabio: bufio.(*Writer).WriteString(0x0, 0xc002df, 0x19, 0x2, 0x1, 0xc1555edc18)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/bufio/bufio.go:688 +0x7f
May 15 10:21:05 <hostname> fabio: net/http.(*expectContinueReader).Read(0xc06e2185d0, 0xc05df1b000, 0xfe, 0x1000, 0x0, 0x0, 0x0)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/net/http/server.go:893 +0x160
May 15 10:21:05 <hostname> fabio: net/http.transferBodyReader.Read(0xc16812edc0, 0xc05df1b000, 0xfe, 0x1000, 0xb40040, 0xc094374001, 0x0)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/net/http/transfer.go:62 +0x56
May 15 10:21:05 <hostname> fabio: io.(*LimitedReader).Read(0xc0d9705c60, 0xc05df1b000, 0x1000, 0x1000, 0x0, 0x40bc00, 0xc0d9705c60)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/io/io.go:448 +0x63
May 15 10:21:05 <hostname> fabio: bufio.(*Writer).ReadFrom(0xc1cbe79400, 0xce8080, 0xc0d9705c60, 0x7f9874955150, 0xc1cbe79400, 0x1201d01)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/bufio/bufio.go:722 +0xe5
May 15 10:21:05 <hostname> fabio: io.copyBuffer(0xce70e0, 0xc1cbe79400, 0xce8080, 0xc0d9705c60, 0x0, 0x0, 0x0, 0x3, 0x1, 0xc1555edd01)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/io/io.go:388 +0x2fc
May 15 10:21:05 <hostname> fabio: io.Copy(...)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/io/io.go:364
May 15 10:21:05 <hostname> fabio: net/http.(*transferWriter).writeBody(0xc16812edc0, 0xce70e0, 0xc1cbe79400, 0x2, 0x2)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/net/http/transfer.go:369 +0x715
May 15 10:21:05 <hostname> fabio: net/http.(*Request).write(0xc0534f3700, 0xce70e0, 0xc1cbe79400, 0x0, 0xc06e218690, 0xc0a42715c0, 0x0, 0x0)
May 15 10:21:05 <hostname> fabio: /usr/local/Cellar/go/1.12.5/libexec/src/net/http/request.go:655 +0x721
May 15 10:21:05 <hostname> fabio: net/http.(*persistConn).writeLoop(0xc018d9c120)

We believe that this is related to the following golang issue:
golang/go#30580

It appears that the panic happens when concurrent expect 100 continue header are received in a client request.

[murphymj25]

@galen0624

[murphymj25]

We have identified a workaround by updating the net/http server.go to recover from the panic:

func (ecr *expectContinueReader) Read(p []byte) (n int, err error) {
	ecr.resp.conn.mu.Lock()
	defer ecr.resp.conn.mu.Unlock()

	defer func() {
		if r := recover(); r != nil {
			log.Println(map[string]interface{}{
				"Recovered": r,
			}, "Recovered from HTTP/S Proxy panic")
		}
	}()

	if ecr.closed {
		return 0, ErrBodyReadAfterClose
	}
@@ -900,9 +907,18 @@ func (ecr *expectContinueReader) Read(p []byte) (n int, err error) {
}

func (ecr *expectContinueReader) Close() error {
	ecr.resp.conn.mu.Lock()

	defer func() {
		if r := recover(); r != nil {
			log.Println(map[string]interface{}{
				"Recovered": r,
			}, "Recovered from HTTP/S Proxy panic")
		}
	}()


	ecr.closed = true
	ecr.resp.conn.mu.Unlock()

	return ecr.readCloser.Close()
}