Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault during high load #789

Open
tobbe76 opened this issue Oct 28, 2024 · 0 comments
Open

Segmentation fault during high load #789

tobbe76 opened this issue Oct 28, 2024 · 0 comments

Comments

@tobbe76
Copy link
Contributor

tobbe76 commented Oct 28, 2024

During high load bazel-remote sometimes(every second day) crash, i believe this due to #771. Buffer is returned to early in some cases.
If "recvResult" returns error then we may exit before the go func() handling s.cache.Put.

We switched back to use #770 instead of #771 and has been running for over 2weeks now without any crashes.

Log:
2024/10/07 09:15:59 Failed to mark /data/cas.v2/29/295c2c111a05828bce5f4a7844986e2b02f4977125f8a861f22f5607b6d9e5f8-1795516-119705455 as complete: chmod /data/cas.v2/29/295c2c111a05828bce5f4a7844986e2b02f4977125f8a861f22f5607b6d9e5f8-1795516-119705455: no such file or directory
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xec pc=0x795c0a]

goroutine 773401358 [running]:
github.com/klauspost/compress/zstd.(*blockDec).decodeBuf(0x0, 0xc5e507f020)
external/com_github_klauspost_compress/zstd/blockdec.go:232 +0x2a
github.com/klauspost/compress/zstd.(*Decoder).nextBlockSync(0xc4288d6300)
external/com_github_klauspost_compress/zstd/decoder.go:508 +0x4f5
github.com/klauspost/compress/zstd.(*Decoder).nextBlock(0xc4288d6300, 0x0?)
external/com_github_klauspost_compress/zstd/decoder.go:420 +0x49
github.com/klauspost/compress/zstd.(*Decoder).Read(0xc4288d6300, {0xcfc80d2000?, 0xc7d3315000?, 0x0?})
external/com_github_klauspost_compress/zstd/decoder.go:145 +0x13c
github.com/klauspost/compress/zstd.closeWrapper.Read({0xc5a18db7e8?}, {0xcfc80d2000?, 0x4f9051?, 0xc7b8ab9b00?})
external/com_github_klauspost_compress/zstd/decoder.go:616 +0x18
io.ReadAtLeast({0x7f77761b2e88, 0xc7b8aed170}, {0xcfc80d2000, 0x100000, 0x100000}, 0x100000)
GOROOT/src/io/io.go:335 +0x90
io.ReadFull(...)
GOROOT/src/io/io.go:354
github.com/buchgr/bazel-remote/v2/cache/disk/casblob.WriteAndClose({0x12a8a58, 0x19fae40}, {0x7f77761b2e88, 0xc7b8aed170}, 0xc7a9a74ab0, 0x1, {0xc78519f873, 0x40}, 0x21976)
cache/disk/casblob/casblob.go:628 +0x7a7
github.com/buchgr/bazel-remote/v2/cache/disk.(*diskCache).writeAndCloseFile(0xc7b9054660?, {0x12a8358, 0xc8bc114210}, {0x7f77761b2e88?, 0xc7b8aed170?}, 0xc8bc115770?, {0xc78519f873?, 0x33?}, 0x21976, 0xc7a9a74ab0)
cache/disk/disk.go:365 +0x114
github.com/buchgr/bazel-remote/v2/cache/disk.(*diskCache).Put(0xc00033e400, {0x12a8358, 0xc8bc114210}, 0x1, {0xc78519f873, 0x40}, 0x21976, {0x7f77761b2e88, 0xc7b8aed170})
cache/disk/disk.go:328 +0x6e7
github.com/buchgr/bazel-remote/v2/server.(*grpcServer).Write.func2.1()
server/grpc_bytestream.go:456 +0xc9
created by github.com/buchgr/bazel-remote/v2/server.(*grpcServer).Write.func2 in goroutine 773389695
server/grpc_bytestream.go:455 +0x13f8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant