panic: runtime error: invalid memory address or nil pointer dereference

[hn0pw]

At the moment we have a lot f restarts of the centrifugo ssl service (~every 5mins)
Version: 1.6.3 on Ubuntu 16.04.1 LTS

In the syslog i got the following messages:

Jan 29 11:27:23 ws centrifugo[4223]: panic: runtime error: invalid memory address or nil pointer dereference
Jan 29 11:27:23 ws centrifugo[4223]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5cb4a5]
Jan 29 11:27:23 ws centrifugo[4223]: goroutine 36161527 [running]:
Jan 29 11:27:23 ws centrifugo[4223]: panic(0x904a00, 0xc420010070)
Jan 29 11:27:23 ws centrifugo[4223]: #011/home/travis/.gimme/versions/go1.7.4.linux.amd64/src/runtime/panic.go:500 +0x1a1
Jan 29 11:27:23 ws centrifugo[4223]: net/http.(*http2responseWriter).CloseNotify.func1(0xc4546dd180, 0xc51dffb5e0)
Jan 29 11:27:23 ws centrifugo[4223]: #011/home/travis/.gimme/versions/go1.7.4.linux.amd64/src/net/http/h2_bundle.go:4700 +0x25
Jan 29 11:27:23 ws centrifugo[4223]: created by net/http.(*http2responseWriter).CloseNotify
Jan 29 11:27:23 ws centrifugo[4223]: #011/home/travis/.gimme/versions/go1.7.4.linux.amd64/src/net/http/h2_bundle.go:4702 +0xda

Because of the repeating restarts, the client connection count is going up:

The first spike to 0 connections was the start i think..

Do you have an idea if it's a problem of centrifugo or system configuration?

[FZambia]

@ramon-ga this looks like Go issue as panic happens inside HTTP/2 implementation.

See golang/go#17868 - this commit most probably solves panic - but it have not been included into recent 1.7.5 for some reason...

[hn0pw]

Yes, because of that i disabled the http2 temporary with GODEBUG="http2server=0" now.
To add this the simple way, add it to /etc/default/centrifugo.
Hopefully with this it did not crash anymore.

Running now for 45min without crashing.

Active: active (running) since Sun 2017-01-29 13:24:38 CET; 45min ago

[FZambia]

@ramon-ga commit 329fe35 should fix it

[FZambia]

@ramon-ga could we wait for a couple of days until Go 1.8 out to make new release? It will contain fixes for this panic so no need to use x/net/http2 lib

[hn0pw]

Wow, you're fast!
From my side this is ok. With disabling the http2 server we're up since 21h, so we have a temporary workaround for the problem.

[FZambia]

@ramon-ga released v1.6.4 using Go 1.7.5, new release must fix this issue - feel free to reopen in case it does not.