crash in Go runtime after port_getn returned EINVAL #82958

davepacheco · 2022-06-15T21:01:17Z

Background: we've got a test suite that spins up single-node CockroachDB clusters a lot of times during each run. We're tracking a few cases where CockroachDB seems to crash during startup. I'm filing this issue for oxidecomputer/omicron#1130 because it looks kind of like memory corruption and wanted y'all's input. But I wanted to mention that we also saw oxidecomputer/omicron#1144 and oxidecomputer/omicron#1146. I don't know if they're related. We also filed golang/go#53289 because that one blows up explicitly inside Go.

For this problem, the failure mode is that CockroachDB prints this to stderr:

I220527 18:52:49.994016 1 util/log/flags.go:201  [-] 1  stderr capture started
runtime: port_getn on fd 4 failed (errno=22)
fatal error: runtime: netpoll failed

and then exits.

For context, port_getn is a libc function on Solaris and illumos systems that's analogous to the poll/epoll/kqueue family of APIs.

Some more data:

the full stderr output
a tarball of the whole CockroachDB directory
a cockroach debug zip

We're using:

$ cockroach version
Build Tag:        v21.2.9
Build Time:       2022/04/28 04:02:42
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.10
C Compiler:       gcc 10.3.0
Build Commit ID:  11787edfcfc157a0df951abc34684e4e18b3ef20
Build Type:       release

on helios-1.0.21004 (an illumos distribution).

This is reproducible but not easily. It takes several hours and often hits some other bug instead (that's how we found the ones I mentioned above).

Now, so far this looks like either an OS or Go runtime issue, but we've got reason to suspect memory corruption and wanted to raise this with you all. Go is clearly not expecting to get EINVAL from port_getn. With DTrace, I confirmed that the kernel really is returning EINVAL. I grabbed a core file at that moment and inspected the arguments being passed to the syscall. Everything looks correct except the struct timespec that Go is passing into the kernel, which is:

{
    tv_sec = c001c97500
    tv_nsec = 0xc000240000
}

Based on reading the Go code, I expected this struct to be zero'd. Since tv_nsec is outside the range [0, 1e9), it makes sense that we'd get EINVAL. From the Go code, I don't see how these values should be there.

Here's the stack trace:

fffffc7feb7fee70 libc.so.1`_portfs+0xa()
fffffc7feb7fef28 runtime.asmsysvicall6+0x5a()
fffffc7feb7ffbc0 runtime.netpoll+0xc5()
fffffc7feb7ffce8 runtime.findrunnable+0xf72()
fffffc7feb7ffd50 runtime.schedule+0x2d7()
fffffc7feb7ffd88 runtime.preemptPark+0xb4()
fffffc7feb7fff38 runtime.newstack+0x2ee()
000000c002702598 runtime.morestack+0xa1()
000000c0027026b0 github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).destroy+0x150()
000000c0027026d8 github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).Close+0x74()
000000c002702898 github.com/cockroachdb/cockroach/pkg/storage.MVCCGet+0x29b()
000000c002702978 github.com/cockroachdb/cockroach/pkg/storage.MVCCGetProto+0xd9()
000000c002702a70 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).tryGetOrCreateReplica.func1+0xfd()
000000c002702c00 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).tryGetOrCreateReplica+0xb3b()
000000c002702dd8 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).getOrCreateReplica+0x1f8()
000000c002702e80 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).acquireSplitLock+0xb6()
000000c002702ec8 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).maybeAcquireSplitMergeLock+0xdf()
000000c0027031f0 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage+0x238()
000000c002703238 github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.Batch.Stage-fm+0x4c()
000000c0027032c0 github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter+0x142()
000000c002703398 github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch+0x185()
000000c002703420 github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries+0xc5()
000000c002703b90 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked+0x100d()
000000c002703cd0 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady+0x11c()
000000c002703e78 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady+0x145()
000000c002703ef8 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker+0x2c2()
000000c002703f20 github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker-fm+0x47()
000000c002703fa0 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2+0xf3()
0000000000000000 runtime.goexit+1()

It looks like something has scribbled over the thread's stack where this struct timespec is supposed to be. Those values (0xc001c97500 and 0xc000240000) look like addresses, and they appear to be coming from the Go memory allocator. Here's the first 20 words at each of those addresses:

> 0xc001c97500,0t20/np
0xc001c97500:   
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  
                0                  

> 0xc000240000,0t20/np
0xc000240000:   
                0xfffffc7fba539b90 
                0xc001224040       
                go.itab.*github.com/cockroachdb/pebble/vfs.enospcFS,github.com/cockroachdb/pebble/vfs.FS
                0xc0010f6300       
                0xc0010e8000       
                0x41               
                2                  
                0x10               
                0                  
                0                  
                0                  
                0x4fdf7            
                0x30b              
                0                  
                0                  
                0                  
                0                  
                0                  
                0x50107            
                0x2de

It's this second one that makes me worried that something inside CockroachDB scribbled over the stack. There's more detail in oxidecomputer/omicron#1130 and more detailed notes about how I came to these conclusions in this comment.

I'd be interested to know if this rings a bell for any of you or if you have thoughts on any of the data here!

Jira issue: CRDB-16755

The text was updated successfully, but these errors were encountered:

blathers-crl · 2022-06-15T21:01:20Z

Hello, I am Blathers. I am here to help you get the issue triaged.

It looks like you have not filled out the issue in the format of any of our templates. To best assist you, we advise you to use one of these templates.

I have CC'd a few people who may be able to assist you:

@cockroachdb/storage (found keywords: pebble)
@cockroachdb/kv (found keywords: kv,MVCC)
@cockroachdb/replication (found keywords: Raft)

If we have not gotten back to your issue within a few business days, you can try the following:

Join our community slack channel and ask on #cockroachdb.
Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

blathers-crl · 2022-06-15T21:01:23Z

cc @cockroachdb/replication

nicktrav · 2022-06-15T21:13:34Z

Thanks for this lovely report, @davepacheco. I'm going to move this from KV-repl into Storage, as this looks to be a Pebble thing - which would make more sense as there's a fair amount of manual memory management happening down there (edit: but certainly not anything jumping to mind that would explain scribbling over another thread's stack 😱).

nicktrav · 2022-06-15T23:12:49Z

@davepacheco - what might help narrow this down a bit on our side, if we suspect Pebble (at least from the stacks you provided and the fact that we are doing some memory management in that library), would be to run our Pebble metamorphic test suite (think of it like our fuzzer for the storage engine).

If you check out Pebble at c50a066abbd3 (which corresponds to the Pebble version used in v21.2.9), and run make stressmeta it will start a burn through an (inifinite) sequence of DB operations and configurations. I'm hoping this gets us to a panic much sooner. It will also print a test seed that can then be used to re-run the same DB configuration / test sequence with something like the following:

$ go test -mod=vendor -tags invariants -v -run TestMeta$ ./internal/metamorphic -seed=123 -keep

In the background I'm going to try and spin up an Illumos VM somewhere and do the same. Hopefully the kernel version, C compiler and Go version you provided is enough to reproduce.

nicktrav · 2022-06-16T00:06:08Z

Doing a re-read again now of the original issue (I should strive to do that more often), the goroutine that puked wasn't in Pebble-land, it was still up in CockroachDB, and looks like it was busy getting some more stack somewhere in here.

However - I was seeing Pebble close to the memory address you mentioned 0xc000240000, and that got me excited.

Pebble may be a red herring, though I would be interested in seeing if you can get the metamorphic tests running on your distribution and whether anything shakes out there. We've had a lot of luck internally finding obscure bugs.

I'm going to try and spin up an Illumos VM somewhere

On this front, I was able to find an OmniOS distribution I could run, but I don't think that's what I want. If it's possible to (easily) build and run the OS y'all are using (helios?), we can keep poking on our side. Otherwise, we might have to leave this up to the experts to debug on your end, with some input on our side (if it looks like it's a Cockroach thing).

jmpesp · 2022-06-16T00:36:08Z

👋

If you check out Pebble at c50a066abbd3 (which corresponds to the Pebble version used in v21.2.9), and run make stressmeta it will start a burn through an (inifinite) sequence of DB operations and configurations.

I did this on a physical omnios machine:

git clone https://github.com/cockroachdb/pebble
cd pebble
git checkout c50a066abbd3
gmake stressmeta

and saw:

go test -mod=vendor -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run TestMeta$ ./internal/metamorphic
# github.com/cockroachdb/pebble/vfs
vfs/vfs.go:140:18: cannot use defaultFS{} (value of type defaultFS) as type FS in variable declaration:
        defaultFS does not implement FS (missing GetDiskUsage method)
vfs/mem_fs.go:322:12: undefined: errNotEmpty
FAIL    github.com/cockroachdb/pebble/internal/metamorphic [build failed]
FAIL
gmake: *** [Makefile:22: test] Error 2

On this front, I was able to find an OmniOS distribution I could run, but I don't think that's what I want.

It is :) both OmniOS and Helios use illumos.

jmpesp · 2022-06-16T00:46:47Z

This patch gets it working:

diff --git a/vfs/disk_usage_solaris.go b/vfs/disk_usage_solaris.go
new file mode 100644
index 00000000..30da621b
--- /dev/null
+++ b/vfs/disk_usage_solaris.go
@@ -0,0 +1,25 @@
+// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use
+// of this source code is governed by a BSD-style license that can be found in
+// the LICENSE file.
+
+// +build solaris
+
+package vfs
+
+import "golang.org/x/sys/unix"
+
+func (defaultFS) GetDiskUsage(path string) (DiskUsage, error) {
+       stat := unix.Statvfs_t{}
+       if err := unix.Statvfs(path, &stat); err != nil {
+               return DiskUsage{}, err
+       }
+
+       freeBytes := uint64(stat.Bsize) * uint64(stat.Bfree)
+       availBytes := uint64(stat.Bsize) * uint64(stat.Bavail)
+       totalBytes := uint64(stat.Bsize) * uint64(stat.Blocks)
+       return DiskUsage{
+               AvailBytes: availBytes,
+               TotalBytes: totalBytes,
+               UsedBytes:  totalBytes - freeBytes,
+       }, nil
+}
diff --git a/vfs/errors_unix.go b/vfs/errors_unix.go
index 31b4dc74..bbc4ebc2 100644
--- a/vfs/errors_unix.go
+++ b/vfs/errors_unix.go
@@ -2,7 +2,7 @@
 // of this source code is governed by a BSD-style license that can be found in
 // the LICENSE file.
 
-// +build darwin dragonfly freebsd linux openbsd
+// +build darwin dragonfly freebsd linux openbsd solaris
 
 package vfs

nicktrav · 2022-06-16T01:38:45Z

It is :) both OmniOS and Helios use illumos.

Neat. Thanks. Wasn't sure if distros would be wildly different. Will keep poking with your patch. 👍

jmpesp · 2022-06-16T15:15:37Z

Update:

while :;
do
        go test -mod=vendor -tags invariants -v -run TestMeta$ ./internal/metamorphic -keep --ops 10000
        sleep 1
done

has been going strong for hours. I'll keep running it for now.

knz · 2022-06-16T15:46:34Z

Q: is this running with the jemalloc custom allocator? or the base go one?

nicktrav · 2022-06-16T15:47:03Z

Thanks @jmpesp! - I've also had a test run going on a VM for ~12 hours without issue. I will note that I built for solaris, and not illumos (was running into some issues building the Go toolchain).

I've asked around internally about this particular issue. Hoping that it will pique the interest of some more folks.

knz · 2022-06-16T15:50:43Z

My question above refers to the fact that by default cockroachdb integrates jemalloc (see cli/start_jemalloc.go). It would be interesting to compare the result for a build produced with the go build tag stdmalloc.

knz · 2022-06-16T15:55:51Z

Another thing worth looking into: have you tried building cockroachdb with the latest go 1.18 instead? There's a couple of changes in the runtime system that this could pick up.

jmpesp · 2022-06-16T16:49:13Z

Q: is this running with the jemalloc custom allocator? or the base go one?

What you see is what I'm running, I'm not sure what is selected.

My question above refers to the fact that by default cockroachdb integrates jemalloc (see cli/start_jemalloc.go). It would be interesting to compare the result for a build produced with the go build tag stdmalloc.

Can I do this for the metamorphic test?

have you tried building cockroachdb with the latest go 1.18 instead?

My go is go version go1.18.3 solaris/amd64.

knz · 2022-06-16T16:54:43Z

What you see is what I'm running, I'm not sure what is selected.

Presumably you know how you built the cockroach binary? Since we're not providing that for you? That would tell.
The log files suggest that jemalloc is disabled, but I can't be sure.

Can I do this for the metamorphic test?

it would be moot if you knew already that jemalloc is not being used.
If it was, then yes, you can add stdmalloc to the list of tags in -tags.

But I'd be interested to see what behavior you observe on the entire cockroach use case from the top of the issue.

My go is go version go1.18.3 solaris/amd64.

Interesting. For the sake of the experiment, do you get different results when building with 1.17.10?

jmpesp · 2022-06-16T17:39:06Z

Presumably you know how you built the cockroach binary? Since we're not providing that for you? That would tell. The log files suggest that jemalloc is disabled, but I can't be sure.

I responded to the question for what I was doing (pebble's metamorphic test), not the original issue (port_getn related crash), sorry for the confusion. The golang issue I'm looking at was linked to and I came to this issue hoping for something related to that which would help with the debugging effort. I'll bow out :)

davepacheco · 2022-06-17T16:12:18Z

Thanks @nicktrav for digging in here! Does @jmpesp's data (that the metamorphic test ran without issue for 12 hours) help?

@knz

Presumably you know how you built the cockroach binary? Since we're not providing that for you? That would tell.
The log files suggest that jemalloc is disabled, but I can't be sure.

Can you tell how we would know, either from the binary or the build process? (It seems like this would be a good addition to the cockroach version output.) As far as I can tell, we're not doing anything to disable jemalloc during the build, and it looks to be included by default. The binary has symbols that look related to jemalloc, but as I assume it would be statically linked, I'm not sure how to tell if these are coming from jemalloc or from CockroachDB symbols that are always included. Examples:

> ::nm ! grep -i jemalloc
0x0000000004e6ea60|0x00000000000002d2|FUNC |LOCL |0x2  |15      |je_jemalloc_postfork_parent
0x0000000004e6e780|0x00000000000002d2|FUNC |LOCL |0x2  |15      |je_jemalloc_postfork_child
0x0000000004e6ed40|0x0000000000000622|FUNC |LOCL |0x2  |15      |je_jemalloc_prefork
0x000000000442fb80|0x00000000000000b6|FUNC |LOCL |0x0  |15      |github.com/cockroachdb/cockroach/pkg/server/status._C2func_jemalloc_get_stats
0x000000000442fda0|0x00000000000006dd|FUNC |LOCL |0x0  |15      |github.com/cockroachdb/cockroach/pkg/server/status.getJemallocStats
0x00000000045bd5a0|0x000000000000016f|FUNC |LOCL |0x0  |15      |github.com/cockroachdb/cockroach/pkg/server/heapprofiler.takeJemallocProfile
0x0000000004d979c0|0x0000000000000165|FUNC |LOCL |0x0  |15      |github.com/cockroachdb/cockroach/pkg/cli.writeJemallocProfile
0x0000000004daf920|0x0000000000000067|FUNC |LOCL |0x0  |15      |github.com/cockroachdb/cockroach/pkg/cli.writeJemallocProfile.func1.1
0x0000000004daf9a0|0x0000000000000074|FUNC |LOCL |0x0  |15      |github.com/cockroachdb/cockroach/pkg/cli.writeJemallocProfile.func1
0x0000000006efa140|0x0000000000000018|OBJT |LOCL |0x0  |18      |github.com/cockroachdb/cockroach/pkg/cli.writeJemallocProfile.stkobj
0x00000000091a6278|0x0000000000000008|OBJT |LOCL |0x0  |28      |github.com/cockroachdb/cockroach/pkg/server/status._cgo_a256212ac815_C2func_jemalloc_get_stats
0x0000000006f6aa80|0x0000000000000028|OBJT |LOCL |0x0  |18      |github.com/cockroachdb/cockroach/pkg/server/status._C2func_jemalloc_get_stats.stkobj
0x0000000006fc1b00|0x0000000000000038|OBJT |LOCL |0x0  |18      |github.com/cockroachdb/cockroach/pkg/server/status.getJemallocStats.stkobj
0x000000000954b428|0x0000000000000008|OBJT |LOCL |0x0  |36      |github.com/cockroachdb/cockroach/pkg/server/heapprofiler.jemallocHeapDump
0x0000000006f06dc0|0x0000000000000018|OBJT |LOCL |0x0  |18      |github.com/cockroachdb/cockroach/pkg/server/heapprofiler.takeJemallocProfile.stkobj
0x0000000000000000|0x0000000000000000|FILE |LOCL |0x0  |ABS     |start_jemalloc.cgo2.c
0x0000000000000000|0x0000000000000000|FILE |LOCL |0x0  |ABS     |runtime_jemalloc.cgo2.c
0x0000000000000000|0x0000000000000000|FILE |LOCL |0x0  |ABS     |jemalloc.c
0x0000000004e7e4a1|0x0000000000000005|FUNC |LOCL |0x0  |15      |je_jemalloc_postfork_child.cold
0x0000000004e7e4a6|0x0000000000000005|FUNC |LOCL |0x0  |15      |je_jemalloc_postfork_parent.cold
0x0000000004e7e4ab|0x0000000000000005|FUNC |LOCL |0x0  |15      |je_jemalloc_prefork.cold
0x0000000004e7e510|0x00000000000000bc|FUNC |LOCL |0x0  |15      |jemalloc_constructor
0x0000000004e7e505|0x0000000000000005|FUNC |LOCL |0x0  |15      |jemalloc_constructor.cold
0x0000000004db6b20|0x0000000000000129|FUNC |GLOB |0x0  |15      |jemalloc_get_stats
0x0000000004db6cc0|0x000000000000003b|FUNC |GLOB |0x0  |15      |_cgo_a256212ac815_Cfunc_jemalloc_get_stats
0x0000000004db6c50|0x0000000000000051|FUNC |GLOB |0x0  |15      |_cgo_a256212ac815_C2func_jemalloc_get_stats

The other reason I think jemalloc is being used is that I tried to LD_PRELOAD libumem.so (which implements malloc(3c) and friends and has good facilities for identifying and debugging corruption), and I ran cockroach and checked with a debugger, and libumem reported zero allocations. So it seems like cockroach is never calling malloc(3c). Does that seem right if jemalloc is being used?

Does a stdmalloc build cause CockroachDB to use malloc(3c) directly? If so, I will probably try that so we can see if libumem can shed some light here.

I have another data point that could be related, but I'm not sure: I just ran into another SIGSEGV: oxidecomputer/omicron#1223. This one looks more obviously inside CockroachDB. I've saved the entire CockroachDB data directory (attached to that ticket), including the full stderr capture. Let me know if there's more you'd like from here. Unfortunately since Go just exits on SIGSEGV, I don't have a core file.

I also wanted to mention that process that triggered this initial report remains stopped on my system at the same point that I mentioned above. I have a core file, a list of open files, arguments, environment, etc. If there's anything else you'd like from the running process, let me know. Otherwise I may kill it soon.

knz · 2022-06-19T09:21:20Z

Does a stdmalloc build cause CockroachDB to use malloc(3c) directly? If so, I will probably try that so we can see if libumem can shed some light here.

Yes please.

I just ran into another SIGSEGV: oxidecomputer/omicron#1223

I replied on that ticket.

that process that triggered this initial report remains stopped on my system at the same point that I mentioned above. I have a core file, a list of open files, arguments, environment, etc.

Would it be possible to emit a stack dump for all the threads in the process?

knz · 2022-06-19T09:21:40Z

Also let me encourage you to build with go 1.17 or 1.18 instead of 1.16.

nicktrav · 2022-06-20T19:59:29Z

Thanks @nicktrav for digging in here! Does @jmpesp's data (that the metamorphic test ran without issue for 12 hours) help?

It's definitely a useful signal - thank you for your help there.

I was also able to do the same, and without issue. There are a couple caveats here in that I'm not sure how similar the environments / binaries are - I tried my best to align the Go runtime, Pebble version and C compiler, but I was running on OmniOS, so there are likely differences in the kernel.

The other thing is that it looks like the issue isn't actually in Pebble itself - it's in adjacent code that has probably called into Pebble (or is just about to), but it's panic-ing above Pebble - so it may not be as interesting that the metamorphic tests aren't picking anything up as we're not exercising the exact codenpaths. That said, we're certainly exercising a lot of the manual memory management code paths, without issue.

As a side note, it probably makes sense for us to build and test Pebble on Solaris / illumos. I think that's tangential to this issue though, but we'll see what we can do.

github-actions · 2023-12-18T11:04:36Z

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

davepacheco · 2023-12-18T21:18:40Z

We haven't seen this problem in a while, though we have no reason to believe it's fixed.

davepacheco added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Jun 15, 2022

blathers-crl bot added A-kv-replication Relating to Raft, consensus, and coordination. O-community Originated from the community X-blathers-triaged blathers was able to find an owner T-kv-replication labels Jun 15, 2022

davepacheco mentioned this issue Jun 15, 2022

cockroachdb crashed in Go runtime after reporting EINVAL from port_getn oxidecomputer/omicron#1130

Closed

nicktrav added A-storage Relating to our storage engine (Pebble) on-disk storage. T-storage Storage Team and removed A-kv-replication Relating to Raft, consensus, and coordination. T-kv-replication labels Jun 15, 2022

nicktrav added T-server-and-security DB Server & Security and removed A-storage Relating to our storage engine (Pebble) on-disk storage. T-storage Storage Team labels Jun 22, 2022

This was referenced Jul 1, 2022

cli: logconfig reset panic due to race condition in cockroach version #83706

Closed

cockroach version panic in schema change #83864

Closed

davepacheco mentioned this issue Nov 22, 2022

crash with GODEBUG=cgocheck=2: Go pointer stored into non-Go memory #92361

Open

github-actions bot added the no-issue-activity label Dec 18, 2023

nicktrav removed the no-issue-activity label Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crash in Go runtime after port_getn returned EINVAL #82958

crash in Go runtime after port_getn returned EINVAL #82958

davepacheco commented Jun 15, 2022 •

edited by cockroach-jira-scripts

Loading

blathers-crl bot commented Jun 15, 2022

blathers-crl bot commented Jun 15, 2022

nicktrav commented Jun 15, 2022 •

edited

Loading

nicktrav commented Jun 15, 2022

nicktrav commented Jun 16, 2022 •

edited

Loading

jmpesp commented Jun 16, 2022

jmpesp commented Jun 16, 2022

nicktrav commented Jun 16, 2022

jmpesp commented Jun 16, 2022

knz commented Jun 16, 2022

nicktrav commented Jun 16, 2022

knz commented Jun 16, 2022

knz commented Jun 16, 2022

jmpesp commented Jun 16, 2022

knz commented Jun 16, 2022

jmpesp commented Jun 16, 2022

davepacheco commented Jun 17, 2022

knz commented Jun 19, 2022

knz commented Jun 19, 2022

nicktrav commented Jun 20, 2022

github-actions bot commented Dec 18, 2023

davepacheco commented Dec 18, 2023

crash in Go runtime after port_getn returned EINVAL #82958

crash in Go runtime after port_getn returned EINVAL #82958

Comments

davepacheco commented Jun 15, 2022 • edited by cockroach-jira-scripts Loading

blathers-crl bot commented Jun 15, 2022

blathers-crl bot commented Jun 15, 2022

nicktrav commented Jun 15, 2022 • edited Loading

nicktrav commented Jun 15, 2022

nicktrav commented Jun 16, 2022 • edited Loading

jmpesp commented Jun 16, 2022

jmpesp commented Jun 16, 2022

nicktrav commented Jun 16, 2022

jmpesp commented Jun 16, 2022

knz commented Jun 16, 2022

nicktrav commented Jun 16, 2022

knz commented Jun 16, 2022

knz commented Jun 16, 2022

jmpesp commented Jun 16, 2022

knz commented Jun 16, 2022

jmpesp commented Jun 16, 2022

davepacheco commented Jun 17, 2022

knz commented Jun 19, 2022

knz commented Jun 19, 2022

nicktrav commented Jun 20, 2022

github-actions bot commented Dec 18, 2023

davepacheco commented Dec 18, 2023

davepacheco commented Jun 15, 2022 •

edited by cockroach-jira-scripts

Loading

nicktrav commented Jun 15, 2022 •

edited

Loading

nicktrav commented Jun 16, 2022 •

edited

Loading