Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory safety issue in backend #55

Closed
zombiezen opened this issue Sep 25, 2024 · 2 comments
Closed

Memory safety issue in backend #55

zombiezen opened this issue Sep 25, 2024 · 2 comments
Labels
backend Build server bug Something isn't working
Milestone

Comments

@zombiezen
Copy link
Collaborator

Every so often, I'm hitting a crash in zb serve due to a bad pointer. This is probably a bug in SQLite, but recording here so I can track down the issue.

Crash log attached from 761ff7e (with some local modifications while hacking on #41).

Crash log
runtime: marked free object in span 0x7f00316c26c8, elemsize=16 freeindex=139 (bad use of unsafe.Pointer? try -d=checkptr)
0xc000458000 alloc marked  
0xc000458010 alloc marked  
0xc000458020 alloc marked  
0xc000458030 alloc unmarked
0xc000458040 alloc unmarked
0xc000458050 alloc unmarked
0xc000458060 alloc unmarked
0xc000458070 alloc unmarked
0xc000458080 alloc unmarked
0xc000458090 alloc unmarked
0xc0004580a0 alloc unmarked
0xc0004580b0 alloc unmarked
0xc0004580c0 alloc unmarked
0xc0004580d0 alloc unmarked
0xc0004580e0 alloc unmarked
0xc0004580f0 alloc unmarked
0xc000458100 alloc unmarked
0xc000458110 alloc unmarked
0xc000458120 alloc unmarked
0xc000458130 alloc unmarked
0xc000458140 alloc unmarked
0xc000458150 alloc unmarked
0xc000458160 alloc unmarked
0xc000458170 alloc unmarked
0xc000458180 alloc unmarked
0xc000458190 alloc unmarked
0xc0004581a0 alloc unmarked
0xc0004581b0 alloc unmarked
0xc0004581c0 alloc unmarked
0xc0004581d0 alloc unmarked
0xc0004581e0 alloc unmarked
0xc0004581f0 alloc unmarked
0xc000458200 alloc unmarked
0xc000458210 alloc unmarked
0xc000458220 alloc unmarked
0xc000458230 alloc unmarked
0xc000458240 alloc unmarked
0xc000458250 alloc unmarked
0xc000458260 alloc unmarked
0xc000458270 alloc unmarked
0xc000458280 alloc unmarked
0xc000458290 alloc marked  
0xc0004582a0 alloc unmarked
0xc0004582b0 alloc marked  
0xc0004582c0 alloc unmarked
0xc0004582d0 alloc unmarked
0xc0004582e0 alloc unmarked
0xc0004582f0 alloc unmarked
0xc000458300 alloc unmarked
0xc000458310 alloc unmarked
0xc000458320 alloc unmarked
0xc000458330 alloc unmarked
0xc000458340 alloc unmarked
0xc000458350 alloc unmarked
0xc000458360 alloc unmarked
0xc000458370 alloc marked  
0xc000458380 alloc unmarked
0xc000458390 alloc unmarked
0xc0004583a0 alloc unmarked
0xc0004583b0 alloc unmarked
0xc0004583c0 alloc unmarked
0xc0004583d0 alloc unmarked
0xc0004583e0 alloc unmarked
0xc0004583f0 alloc marked  
0xc000458400 alloc unmarked
0xc000458410 alloc unmarked
0xc000458420 alloc unmarked
0xc000458430 alloc unmarked
0xc000458440 alloc unmarked
0xc000458450 alloc unmarked
0xc000458460 alloc unmarked
0xc000458470 alloc unmarked
0xc000458480 alloc unmarked
0xc000458490 alloc unmarked
0xc0004584a0 alloc unmarked
0xc0004584b0 alloc unmarked
0xc0004584c0 alloc unmarked
0xc0004584d0 alloc unmarked
0xc0004584e0 alloc unmarked
0xc0004584f0 alloc unmarked
0xc000458500 alloc unmarked
0xc000458510 alloc unmarked
0xc000458520 alloc unmarked
0xc000458530 alloc unmarked
0xc000458540 alloc unmarked
0xc000458550 alloc unmarked
0xc000458560 alloc unmarked
0xc000458570 alloc unmarked
0xc000458580 alloc unmarked
0xc000458590 alloc unmarked
0xc0004585a0 alloc unmarked
0xc0004585b0 alloc unmarked
0xc0004585c0 alloc marked  
0xc0004585d0 alloc unmarked
0xc0004585e0 alloc unmarked
0xc0004585f0 alloc unmarked
0xc000458600 alloc marked  
0xc000458610 alloc unmarked
0xc000458620 alloc unmarked
0xc000458630 alloc unmarked
0xc000458640 alloc unmarked
0xc000458650 alloc unmarked
0xc000458660 alloc marked  
0xc000458670 alloc unmarked
0xc000458680 alloc unmarked
0xc000458690 alloc marked  
0xc0004586a0 alloc unmarked
0xc0004586b0 alloc unmarked
0xc0004586c0 alloc unmarked
0xc0004586d0 alloc unmarked
0xc0004586e0 alloc unmarked
0xc0004586f0 alloc unmarked
0xc000458700 alloc unmarked
0xc000458710 alloc unmarked
0xc000458720 alloc unmarked
0xc000458730 alloc unmarked
0xc000458740 alloc unmarked
0xc000458750 alloc unmarked
0xc000458760 alloc unmarked
0xc000458770 alloc unmarked
0xc000458780 alloc unmarked
0xc000458790 alloc unmarked
0xc0004587a0 alloc unmarked
0xc0004587b0 alloc unmarked
0xc0004587c0 alloc unmarked
0xc0004587d0 alloc unmarked
0xc0004587e0 alloc unmarked
0xc0004587f0 alloc unmarked
0xc000458800 alloc unmarked
0xc000458810 alloc unmarked
0xc000458820 alloc unmarked
0xc000458830 alloc unmarked
0xc000458840 alloc unmarked
0xc000458850 alloc unmarked
0xc000458860 alloc unmarked
0xc000458870 alloc unmarked
0xc000458880 alloc unmarked
0xc000458890 alloc unmarked
0xc0004588a0 alloc unmarked
0xc0004588b0 free  unmarked
0xc0004588c0 free  unmarked
0xc0004588d0 free  unmarked
0xc0004588e0 free  unmarked
0xc0004588f0 free  unmarked
0xc000458900 free  unmarked
0xc000458910 free  unmarked
0xc000458920 free  unmarked
0xc000458930 free  unmarked
0xc000458940 free  unmarked
0xc000458950 free  unmarked
0xc000458960 free  unmarked
0xc000458970 free  unmarked
0xc000458980 free  unmarked
0xc000458990 free  unmarked
0xc0004589a0 free  unmarked
0xc0004589b0 free  unmarked
0xc0004589c0 free  unmarked
0xc0004589d0 free  unmarked
0xc0004589e0 free  unmarked
0xc0004589f0 free  unmarked
0xc000458a00 free  unmarked
0xc000458a10 free  unmarked
0xc000458a20 free  unmarked
0xc000458a30 free  unmarked
0xc000458a40 free  unmarked
0xc000458a50 free  unmarked
0xc000458a60 free  unmarked
0xc000458a70 free  unmarked
0xc000458a80 free  unmarked
0xc000458a90 free  unmarked
0xc000458aa0 free  unmarked
0xc000458ab0 free  unmarked
0xc000458ac0 free  unmarked
0xc000458ad0 free  unmarked
0xc000458ae0 free  unmarked
0xc000458af0 free  unmarked
0xc000458b00 free  unmarked
0xc000458b10 free  unmarked
0xc000458b20 free  unmarked
0xc000458b30 free  unmarked
0xc000458b40 free  unmarked
0xc000458b50 free  unmarked
0xc000458b60 free  unmarked
0xc000458b70 free  unmarked
0xc000458b80 free  unmarked
0xc000458b90 free  unmarked
0xc000458ba0 free  unmarked
0xc000458bb0 free  unmarked
0xc000458bc0 free  unmarked
0xc000458bd0 free  unmarked
0xc000458be0 free  unmarked
0xc000458bf0 free  unmarked
0xc000458c00 free  unmarked
0xc000458c10 free  unmarked
0xc000458c20 free  unmarked
0xc000458c30 free  unmarked
0xc000458c40 free  unmarked
0xc000458c50 free  unmarked
0xc000458c60 free  unmarked
0xc000458c70 free  unmarked
0xc000458c80 free  unmarked
0xc000458c90 free  unmarked
0xc000458ca0 free  unmarked
0xc000458cb0 free  unmarked
0xc000458cc0 free  unmarked
0xc000458cd0 free  unmarked
0xc000458ce0 free  unmarked
0xc000458cf0 free  unmarked
0xc000458d00 free  unmarked
0xc000458d10 free  unmarked
0xc000458d20 free  unmarked
0xc000458d30 free  unmarked
0xc000458d40 free  unmarked
0xc000458d50 free  unmarked
0xc000458d60 alloc marked  
0xc000458d70 free  unmarked
0xc000458d80 free  unmarked
0xc000458d90 free  unmarked
0xc000458da0 free  unmarked
0xc000458db0 free  unmarked
0xc000458dc0 free  unmarked
0xc000458dd0 free  unmarked
0xc000458de0 free  unmarked
0xc000458df0 free  unmarked
0xc000458e00 free  unmarked
0xc000458e10 free  unmarked
0xc000458e20 free  unmarked
0xc000458e30 free  unmarked
0xc000458e40 free  unmarked
0xc000458e50 free  unmarked
0xc000458e60 free  unmarked
0xc000458e70 free  unmarked
0xc000458e80 free  unmarked
0xc000458e90 free  unmarked
0xc000458ea0 free  unmarked
0xc000458eb0 free  unmarked
0xc000458ec0 free  unmarked
0xc000458ed0 free  unmarked
0xc000458ee0 free  unmarked
0xc000458ef0 free  unmarked
0xc000458f00 free  unmarked
0xc000458f10 free  unmarked
0xc000458f20 free  unmarked
0xc000458f30 free  unmarked
0xc000458f40 free  unmarked
0xc000458f50 free  unmarked
0xc000458f60 free  unmarked
0xc000458f70 free  unmarked
0xc000458f80 free  unmarked
0xc000458f90 free  unmarked
0xc000458fa0 free  unmarked
0xc000458fb0 free  unmarked
0xc000458fc0 free  unmarked
0xc000458fd0 free  unmarked
0xc000458fe0 free  unmarked
0xc000458ff0 free  unmarked
0xc000459000 free  unmarked
0xc000459010 free  unmarked
0xc000459020 free  unmarked
0xc000459030 free  unmarked
0xc000459040 free  unmarked
0xc000459050 free  unmarked
0xc000459060 free  unmarked
0xc000459070 free  unmarked
0xc000459080 free  unmarked
0xc000459090 free  unmarked
0xc0004590a0 free  unmarked
0xc0004590b0 free  unmarked
0xc0004590c0 free  unmarked
0xc0004590d0 alloc marked  
0xc0004590e0 free  unmarked
0xc0004590f0 free  unmarked
0xc000459100 free  unmarked
0xc000459110 free  unmarked
0xc000459120 free  unmarked
0xc000459130 free  unmarked
0xc000459140 free  unmarked
0xc000459150 free  unmarked
0xc000459160 free  unmarked
0xc000459170 free  unmarked
0xc000459180 free  marked   zombie
0x000000c000459180:  0x000000c0003b8dc0  0x0000000000000033 
0xc000459190 free  unmarked
0xc0004591a0 free  unmarked
0xc0004591b0 free  unmarked
0xc0004591c0 free  unmarked
0xc0004591d0 free  unmarked
0xc0004591e0 free  unmarked
0xc0004591f0 free  unmarked
0xc000459200 free  unmarked
0xc000459210 free  unmarked
0xc000459220 free  unmarked
0xc000459230 free  unmarked
0xc000459240 free  unmarked
0xc000459250 free  unmarked
0xc000459260 free  unmarked
0xc000459270 free  unmarked
0xc000459280 free  unmarked
0xc000459290 free  unmarked
0xc0004592a0 free  unmarked
0xc0004592b0 free  unmarked
0xc0004592c0 free  unmarked
0xc0004592d0 free  unmarked
0xc0004592e0 free  unmarked
0xc0004592f0 free  unmarked
0xc000459300 free  unmarked
0xc000459310 free  unmarked
0xc000459320 free  unmarked
0xc000459330 free  unmarked
0xc000459340 free  unmarked
0xc000459350 free  unmarked
0xc000459360 free  unmarked
0xc000459370 free  unmarked
0xc000459380 free  unmarked
0xc000459390 free  unmarked
0xc0004593a0 free  unmarked
0xc0004593b0 free  unmarked
0xc0004593c0 free  unmarked
0xc0004593d0 free  unmarked
0xc0004593e0 free  unmarked
0xc0004593f0 free  unmarked
0xc000459400 free  unmarked
0xc000459410 free  unmarked
0xc000459420 free  unmarked
0xc000459430 free  unmarked
0xc000459440 free  unmarked
0xc000459450 free  unmarked
0xc000459460 free  unmarked
0xc000459470 free  unmarked
0xc000459480 free  unmarked
0xc000459490 free  unmarked
0xc0004594a0 free  unmarked
0xc0004594b0 free  unmarked
0xc0004594c0 free  unmarked
0xc0004594d0 free  unmarked
0xc0004594e0 free  unmarked
0xc0004594f0 free  unmarked
0xc000459500 free  unmarked
0xc000459510 free  unmarked
0xc000459520 free  unmarked
0xc000459530 free  unmarked
0xc000459540 free  unmarked
0xc000459550 free  unmarked
0xc000459560 free  unmarked
0xc000459570 free  unmarked
0xc000459580 free  unmarked
0xc000459590 free  unmarked
0xc0004595a0 free  unmarked
0xc0004595b0 free  unmarked
0xc0004595c0 free  unmarked
0xc0004595d0 free  unmarked
0xc0004595e0 free  unmarked
0xc0004595f0 free  unmarked
0xc000459600 free  unmarked
0xc000459610 free  unmarked
0xc000459620 free  unmarked
0xc000459630 free  unmarked
0xc000459640 free  unmarked
0xc000459650 free  unmarked
0xc000459660 free  unmarked
0xc000459670 free  unmarked
0xc000459680 free  unmarked
0xc000459690 free  unmarked
0xc0004596a0 free  unmarked
0xc0004596b0 free  unmarked
0xc0004596c0 free  unmarked
0xc0004596d0 free  unmarked
0xc0004596e0 free  unmarked
0xc0004596f0 free  unmarked
0xc000459700 free  unmarked
0xc000459710 free  unmarked
0xc000459720 free  unmarked
0xc000459730 free  unmarked
0xc000459740 free  unmarked
0xc000459750 free  unmarked
0xc000459760 free  unmarked
0xc000459770 free  unmarked
0xc000459780 free  unmarked
0xc000459790 free  unmarked
0xc0004597a0 free  unmarked
0xc0004597b0 free  unmarked
0xc0004597c0 free  unmarked
0xc0004597d0 free  unmarked
0xc0004597e0 free  unmarked
0xc0004597f0 free  unmarked
0xc000459800 free  unmarked
0xc000459810 free  unmarked
0xc000459820 free  unmarked
0xc000459830 free  unmarked
0xc000459840 free  unmarked
0xc000459850 free  unmarked
0xc000459860 free  unmarked
0xc000459870 free  unmarked
0xc000459880 free  unmarked
0xc000459890 free  unmarked
0xc0004598a0 free  unmarked
0xc0004598b0 free  unmarked
0xc0004598c0 free  unmarked
0xc0004598d0 free  unmarked
0xc0004598e0 free  unmarked
0xc0004598f0 free  unmarked
0xc000459900 free  unmarked
0xc000459910 free  unmarked
0xc000459920 free  unmarked
0xc000459930 free  unmarked
0xc000459940 free  unmarked
0xc000459950 free  unmarked
0xc000459960 free  unmarked
0xc000459970 free  unmarked
0xc000459980 free  unmarked
0xc000459990 free  unmarked
0xc0004599a0 free  unmarked
0xc0004599b0 free  unmarked
0xc0004599c0 free  unmarked
0xc0004599d0 free  unmarked
0xc0004599e0 free  unmarked
0xc0004599f0 free  unmarked
0xc000459a00 free  unmarked
0xc000459a10 free  unmarked
0xc000459a20 free  unmarked
0xc000459a30 free  unmarked
0xc000459a40 free  unmarked
0xc000459a50 free  unmarked
0xc000459a60 free  unmarked
0xc000459a70 free  unmarked
0xc000459a80 free  unmarked
0xc000459a90 free  unmarked
0xc000459aa0 free  unmarked
0xc000459ab0 free  unmarked
0xc000459ac0 free  unmarked
0xc000459ad0 free  unmarked
0xc000459ae0 free  unmarked
0xc000459af0 free  unmarked
0xc000459b00 free  unmarked
0xc000459b10 free  unmarked
0xc000459b20 free  unmarked
0xc000459b30 free  unmarked
0xc000459b40 free  unmarked
0xc000459b50 free  unmarked
0xc000459b60 free  unmarked
0xc000459b70 free  unmarked
0xc000459b80 free  unmarked
0xc000459b90 free  unmarked
0xc000459ba0 free  unmarked
0xc000459bb0 free  unmarked
0xc000459bc0 free  unmarked
0xc000459bd0 free  unmarked
0xc000459be0 free  unmarked
0xc000459bf0 free  unmarked
0xc000459c00 free  unmarked
0xc000459c10 free  unmarked
0xc000459c20 free  unmarked
0xc000459c30 free  unmarked
0xc000459c40 free  unmarked
0xc000459c50 free  unmarked
0xc000459c60 free  unmarked
0xc000459c70 free  unmarked
0xc000459c80 free  unmarked
0xc000459c90 alloc marked  
0xc000459ca0 free  unmarked
0xc000459cb0 free  unmarked
0xc000459cc0 free  unmarked
0xc000459cd0 free  unmarked
0xc000459ce0 free  unmarked
0xc000459cf0 free  unmarked
0xc000459d00 free  unmarked
0xc000459d10 free  unmarked
0xc000459d20 free  unmarked
0xc000459d30 free  unmarked
0xc000459d40 free  unmarked
0xc000459d50 free  unmarked
0xc000459d60 free  unmarked
0xc000459d70 free  unmarked
0xc000459d80 free  unmarked
0xc000459d90 free  unmarked
0xc000459da0 free  unmarked
0xc000459db0 free  unmarked
0xc000459dc0 free  unmarked
0xc000459dd0 free  unmarked
0xc000459de0 free  unmarked
0xc000459df0 free  unmarked
0xc000459e00 free  unmarked
0xc000459e10 free  unmarked
0xc000459e20 free  unmarked
0xc000459e30 free  unmarked
0xc000459e40 free  unmarked
0xc000459e50 free  unmarked
0xc000459e60 free  unmarked
0xc000459e70 free  unmarked
0xc000459e80 free  unmarked
0xc000459e90 free  unmarked
0xc000459ea0 free  unmarked
0xc000459eb0 free  unmarked
0xc000459ec0 free  unmarked
0xc000459ed0 free  unmarked
0xc000459ee0 free  unmarked
0xc000459ef0 free  unmarked
0xc000459f00 free  unmarked
0xc000459f10 free  unmarked
0xc000459f20 free  unmarked
0xc000459f30 free  unmarked
0xc000459f40 free  unmarked
0xc000459f50 free  unmarked
0xc000459f60 free  unmarked
0xc000459f70 free  unmarked
fatal error: found pointer to free object

runtime stack:
runtime.throw({0xb53943?, 0xc000459190?})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/panic.go:1067 +0x48 fp=0x7f0032a6ea88 sp=0x7f0032a6ea58 pc=0x4733a8
runtime.(*mspan).reportZombies(0x7f00316c26c8)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcsweep.go:890 +0x2ea fp=0x7f0032a6eb08 sp=0x7f0032a6ea88 pc=0x42b42a
runtime.(*sweepLocked).sweep(0x0?, 0x0)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcsweep.go:658 +0xb54 fp=0x7f0032a6ec28 sp=0x7f0032a6eb08 pc=0x42aa74
runtime.(*mcentral).uncacheSpan(0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mcentral.go:236 +0x98 fp=0x7f0032a6ec50 sp=0x7f0032a6ec28 pc=0x41a8f8
runtime.(*mcache).releaseAll(0x7f007ad44108)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mcache.go:291 +0x13e fp=0x7f0032a6ecb8 sp=0x7f0032a6ec50 pc=0x41a25e
runtime.(*mcache).prepareForSweep(0x7f007ad44108)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mcache.go:328 +0x35 fp=0x7f0032a6ece0 sp=0x7f0032a6ecb8 pc=0x41a355
runtime.procresize(0x10)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:5751 +0x537 fp=0x7f0032a6edb0 sp=0x7f0032a6ece0 pc=0x44a957
runtime.startTheWorldWithSema(0xc081a6aef38, {0x60?, 0x47d380?, 0x33ac?, 0x382d3757?})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:1675 +0xa5 fp=0x7f0032a6ee18 sp=0x7f0032a6edb0 pc=0x442085
runtime.gcMarkTermination.func3()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1126 +0x27 fp=0x7f0032a6ee50 sp=0x7f0032a6ee18 pc=0x420047
runtime.systemstack(0x800000)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:514 +0x4a fp=0x7f0032a6ee60 sp=0x7f0032a6ee50 pc=0x47940a

goroutine 77 gp=0xc0000e5c00 m=5 mp=0xc000100008 [running]:
runtime.systemstack_switch()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:479 +0x8 fp=0xc00050ec70 sp=0xc00050ec60 pc=0x4793a8
runtime.gcMarkTermination({0x20?, 0xc081a6a6997?, 0x10?, 0x0?})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1116 +0x545 fp=0xc00050ee98 sp=0xc00050ec70 pc=0x41f6c5
runtime.gcMarkDone()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:954 +0x31e fp=0xc00050ef38 sp=0xc00050ee98 pc=0x41eefe
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1499 +0x331 fp=0xc00050efc8 sp=0xc00050ef38 pc=0x420671
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00050efe0 sp=0xc00050efc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050efe8 sp=0xc00050efe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 1 gp=0xc0000061c0 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0000f5750 sp=0xc0000f5730 pc=0x4734ce
runtime.netpollblock(0x41a9c8?, 0x40a8c6?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000f5788 sp=0xc0000f5750 pc=0x437837
internal/poll.runtime_pollWait(0x7f0033ab46c8, 0x72)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:351 +0x85 fp=0xc0000f57a8 sp=0xc0000f5788 pc=0x4727c5
internal/poll.(*pollDesc).wait(0xc00003a180?, 0x0?, 0x0)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000f57d0 sp=0xc0000f57a8 pc=0x4f5e87
internal/poll.(*pollDesc).waitRead(...)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00003a180)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_unix.go:620 +0x295 fp=0xc0000f5878 sp=0xc0000f57d0 pc=0x4fb255
net.(*netFD).accept(0xc00003a180)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/fd_unix.go:172 +0x29 fp=0xc0000f5930 sp=0xc0000f5878 pc=0x5957e9
net.(*UnixListener).accept(0x448a00?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/unixsock_posix.go:172 +0x16 fp=0xc0000f5958 sp=0xc0000f5930 pc=0x5ae596
net.(*UnixListener).AcceptUnix(0xc0000c4300)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/unixsock.go:247 +0x30 fp=0xc0000f5988 sp=0xc0000f5958 pc=0x5acfb0
main.runServe({0xc4d1a8, 0xc0001303c0}, 0xc00012d140, 0xc00012d3b0)
/home/light/src/zb.256lights.llc/pkg/cmd/zb/serve.go:129 +0x61e fp=0xc0000f5b58 sp=0xc0000f5988 pc=0xa2ed1e
main.newServeCommand.func1(0xc000162600?, {0xc00009c1c0?, 0x4?, 0xb46152?})
/home/light/src/zb.256lights.llc/pkg/cmd/zb/serve.go:54 +0x2c fp=0xc0000f5b88 sp=0xc0000f5b58 pc=0xa2e6ac
github.com/spf13/cobra.(*Command).execute(0xc000201508, {0xc00009c180, 0x4, 0x4})
/home/light/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xaaa fp=0xc0000f5d10 sp=0xc0000f5b88 pc=0x604faa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000200308)
/home/light/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff fp=0xc0000f5de8 sp=0xc0000f5d10 pc=0x60587f
github.com/spf13/cobra.(*Command).Execute(...)
/home/light/pkg/mod/github.com/spf13/[email protected]/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(...)
/home/light/pkg/mod/github.com/spf13/[email protected]/command.go:1032
main.main()
/home/light/src/zb.256lights.llc/pkg/cmd/zb/main.go:89 +0x565 fp=0xc0000f5f50 sp=0xc0000f5de8 pc=0xa2cba5
runtime.main()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:272 +0x28b fp=0xc0000f5fe0 sp=0xc0000f5f50 pc=0x43ef0b
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x47b3c1

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00006cfa8 sp=0xc00006cf88 pc=0x4734ce
runtime.goparkunlock(...)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
runtime.forcegchelper()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:337 +0xb3 fp=0xc00006cfe0 sp=0xc00006cfa8 pc=0x43f253
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006cfe8 sp=0xc00006cfe0 pc=0x47b3c1
created by runtime.init.7 in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000007180 m=nil [runnable]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00006d780 sp=0xc00006d760 pc=0x4734ce
runtime.goparkunlock(...)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
runtime.bgsweep(0xc00009a000)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcsweep.go:317 +0xdf fp=0xc00006d7c8 sp=0xc00006d780 pc=0x429a1f
runtime.gcenable.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006d7e0 sp=0xc00006d7c8 pc=0x41e105
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006d7e8 sp=0xc00006d7e0 pc=0x47b3c1
created by runtime.gcenable in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0x5b0884?, 0x54f189?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00006df78 sp=0xc00006df58 pc=0x4734ce
runtime.goparkunlock(...)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x10a49c0)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00006dfa8 sp=0xc00006df78 pc=0x427409
runtime.bgscavenge(0xc00009a000)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00006dfc8 sp=0xc00006dfa8 pc=0x427999
runtime.gcenable.gowrap2()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0x41e0a5
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x47b3c1
created by runtime.gcenable in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc000104700 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0xb9e508?, 0x0?, 0x40?, 0x2000000020?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00006c620 sp=0xc00006c600 pc=0x4734ce
runtime.runfinq()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00006c7e0 sp=0xc00006c620 pc=0x41d187
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006c7e8 sp=0xc00006c7e0 pc=0x47b3c1
created by runtime.createfing in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mfinal.go:163 +0x3d

goroutine 19 gp=0xc0001048c0 m=nil [chan receive]:
runtime.gopark(0xc000068760?, 0xa22325?, 0xd0?, 0xc5?, 0xc55640?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc000068718 sp=0xc0000686f8 pc=0x4734ce
runtime.chanrecv(0xc0001120e0, 0x0, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:639 +0x41c fp=0xc000068790 sp=0xc000068718 pc=0x40d4bc
runtime.chanrecv1(0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:489 +0x12 fp=0xc0000687b8 sp=0xc000068790 pc=0x40d072
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1735 +0x2f fp=0xc0000687e0 sp=0xc0000687b8 pc=0x42110f
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0000687e8 sp=0xc0000687e0 pc=0x47b3c1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1730 +0x96

goroutine 20 gp=0xc000104a80 m=nil [select, locked to thread]:
runtime.gopark(0xc000068fa8?, 0x2?, 0xa8?, 0x8e?, 0xc000068f94?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc000068e30 sp=0xc000068e10 pc=0x4734ce
runtime.selectgo(0xc000068fa8, 0xc000068f90, 0x0?, 0x0, 0x0?, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0x7a5 fp=0xc000068f58 sp=0xc000068e30 pc=0x450de5
runtime.ensureSigM.func1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/signal_unix.go:1060 +0x19f fp=0xc000068fe0 sp=0xc000068f58 pc=0x46adbf
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x47b3c1
created by runtime.ensureSigM in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/signal_unix.go:1043 +0xc8

goroutine 5 gp=0xc000007880 m=3 mp=0xc000072e08 [syscall]:
runtime.notetsleepg(0x10c7bc0, 0xffffffffffffffff)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/lock_futex.go:246 +0x29 fp=0xc00006e7a0 sp=0xc00006e778 pc=0x412cc9
os/signal.signal_recv()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc00006e7c0 sp=0xc00006e7a0 pc=0x4753c9
os/signal.loop()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00006e7e0 sp=0xc00006e7c0 pc=0x6959f3
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x47b3c1
created by os/signal.Notify.func1.1 in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/os/signal/signal.go:151 +0x1f

goroutine 6 gp=0xc000007dc0 m=nil [select]:
runtime.gopark(0xc00006efb0?, 0x2?, 0x68?, 0x4a?, 0xc00006ef9c?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00006ee40 sp=0xc00006ee20 pc=0x4734ce
runtime.selectgo(0xc00006efb0, 0xc00006ef98, 0x0?, 0x0, 0x0?, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0x7a5 fp=0xc00006ef68 sp=0xc00006ee40 pc=0x450de5
os/signal.NotifyContext.func1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/os/signal/signal.go:288 +0x66 fp=0xc00006efe0 sp=0xc00006ef68 pc=0x695586
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x47b3c1
created by os/signal.NotifyContext in goroutine 1
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/os/signal/signal.go:287 +0x156

goroutine 7 gp=0xc0000e4000 m=nil [chan receive]:
runtime.gopark(0xab3600?, 0x0?, 0x90?, 0xf6?, 0x4ad1d9?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00006f600 sp=0xc00006f5e0 pc=0x4734ce
runtime.chanrecv(0xc0001122a0, 0x0, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:639 +0x41c fp=0xc00006f678 sp=0xc00006f600 pc=0x40d4bc
runtime.chanrecv1(0xc0000a60f0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:489 +0x12 fp=0xc00006f6a0 sp=0xc00006f678 pc=0x40d072
main.runServe.func1()
/home/light/src/zb.256lights.llc/pkg/cmd/zb/serve.go:93 +0x99 fp=0xc00006f7e0 sp=0xc00006f6a0 pc=0xa2f619
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006f7e8 sp=0xc00006f7e0 pc=0x47b3c1
created by main.runServe in goroutine 1
/home/light/src/zb.256lights.llc/pkg/cmd/zb/serve.go:89 +0x252

goroutine 63 gp=0xc0000e41c0 m=nil [GC worker (idle)]:
runtime.gopark(0xc0001ae000?, 0xc00019a040?, 0x60?, 0x43?, 0xc000014220?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a6738 sp=0xc0001a6718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a67c8 sp=0xc0001a6738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a67e0 sp=0xc0001a67c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a67e8 sp=0xc0001a67e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 62 gp=0xc0000e4380 m=nil [GC worker (idle)]:
runtime.gopark(0xc0000b3180?, 0xc00006ff98?, 0x1?, 0x0?, 0x2?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00006ff38 sp=0xc00006ff18 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00006ffc8 sp=0xc00006ff38 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 34 gp=0xc000188380 m=nil [chan receive]:
runtime.gopark(0xc000085df0?, 0x46e35d?, 0x0?, 0x2c?, 0xc000085de0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc000085d50 sp=0xc000085d30 pc=0x4734ce
runtime.chanrecv(0xc000182070, 0xc000085e00, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:639 +0x41c fp=0xc000085dc8 sp=0xc000085d50 pc=0x40d4bc
runtime.chanrecv2(0xc000085df8?, 0x448a00?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:494 +0x12 fp=0xc000085df0 sp=0xc000085dc8 pc=0x40d092
zb.256lights.llc/pkg/zbstore.(*Codec).ReadRequest(0xc0001a0040)
/home/light/src/zb.256lights.llc/pkg/zbstore/codec.go:71 +0x33 fp=0xc000085e28 sp=0xc000085df0 pc=0x67ddf3
zb.256lights.llc/pkg/internal/jsonrpc.Serve({0xc4cf78, 0xc0001b2030}, {0xc4a4c0, 0xc0001a0040}, {0xc48720, 0xc0000ee070})
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/server.go:66 +0xfa fp=0xc000085ed0 sp=0xc000085e28 pc=0x61d49a
main.runServe.func4()
/home/light/src/zb.256lights.llc/pkg/cmd/zb/serve.go:150 +0x1de fp=0xc000085fe0 sp=0xc000085ed0 pc=0xa2f13e
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x47b3c1
created by main.runServe in goroutine 1
/home/light/src/zb.256lights.llc/pkg/cmd/zb/serve.go:141 +0x611

goroutine 35 gp=0xc000188540 m=nil [IO wait]:
runtime.gopark(0xd9?, 0x1000?, 0x0?, 0x0?, 0xb?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc000487a10 sp=0xc0004879f0 pc=0x4734ce
runtime.netpollblock(0x494098?, 0x40a8c6?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc000487a48 sp=0xc000487a10 pc=0x437837
internal/poll.runtime_pollWait(0x7f0033ab45c0, 0x72)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/netpoll.go:351 +0x85 fp=0xc000487a68 sp=0xc000487a48 pc=0x4727c5
internal/poll.(*pollDesc).wait(0xc000196080?, 0xc000204000?, 0x0)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000487a90 sp=0xc000487a68 pc=0x4f5e87
internal/poll.(*pollDesc).waitRead(...)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000196080, {0xc000204000, 0x1000, 0x1000})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/internal/poll/fd_unix.go:165 +0x27a fp=0xc000487b28 sp=0xc000487a90 pc=0x4f717a
net.(*netFD).Read(0xc000196080, {0xc000204000?, 0x18?, 0xc000033be0?})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/fd_posix.go:55 +0x25 fp=0xc000487b70 sp=0xc000487b28 pc=0x593825
net.(*conn).Read(0xc00019c000, {0xc000204000?, 0xc00020401d?, 0x2f?})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/net.go:189 +0x45 fp=0xc000487bb8 sp=0xc000487b70 pc=0x59fd85
net.(*UnixConn).Read(0x2f?, {0xc000204000?, 0x13a?, 0xc000033ca8?})
:1 +0x25 fp=0xc000487be8 sp=0xc000487bb8 pc=0x5b2805
main.(*nopCloser).Read(0x7ffffffffffffca1?, {0xc000204000?, 0x0?, 0x7f007ad44108?})
:1 +0x29 fp=0xc000487c18 sp=0xc000487be8 pc=0xa31fe9
bufio.(*Reader).fill(0xc000132480)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bufio/bufio.go:110 +0x103 fp=0xc000487c50 sp=0xc000487c18 pc=0x5b6023
bufio.(*Reader).Peek(0xc000132480, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/bufio/bufio.go:148 +0x53 fp=0xc000487c70 sp=0xc000487c50 pc=0x5b6153
net/textproto.(*Reader).upcomingHeaderKeys(0xc000487df0)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/textproto/reader.go:617 +0x27 fp=0xc000487cb8 sp=0xc000487c70 pc=0x617387
net/textproto.readMIMEHeader(0xc000487df0, 0x7fffffffffffffff, 0x7fffffffffffffff)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/textproto/reader.go:515 +0x37 fp=0xc000487db0 sp=0xc000487cb8 pc=0x617897
net/textproto.(*Reader).ReadMIMEHeader(...)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/net/textproto/reader.go:502
zb.256lights.llc/pkg/internal/jsonrpc.(*Reader).NextMessage(0xc00012d470)
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/frame.go:60 +0x165 fp=0xc000487e38 sp=0xc000487db0 pc=0x61b625
zb.256lights.llc/pkg/zbstore.readLoop(0xc000182070, 0xc0001820e0, {0xc4a498, 0xc0001ae000}, 0xc00012d470)
/home/light/src/zb.256lights.llc/pkg/zbstore/codec.go:89 +0x54 fp=0xc000487ed8 sp=0xc000487e38 pc=0x67df14
zb.256lights.llc/pkg/zbstore.NewCodec.func1()
/home/light/src/zb.256lights.llc/pkg/zbstore/codec.go:64 +0x22c fp=0xc000487fe0 sp=0xc000487ed8 pc=0x67dc8c
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000487fe8 sp=0xc000487fe0 pc=0x47b3c1
created by zb.256lights.llc/pkg/zbstore.NewCodec in goroutine 34
/home/light/src/zb.256lights.llc/pkg/zbstore/codec.go:58 +0x2a5

goroutine 36 gp=0xc000188700 m=nil [select]:
runtime.gopark(0xc0001baca0?, 0x4?, 0x60?, 0x20?, 0xc0001babf0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001baa60 sp=0xc0001baa40 pc=0x4734ce
runtime.selectgo(0xc0001baca0, 0xc0001babe8, 0x2?, 0x0, 0x10c9220?, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0x7a5 fp=0xc0001bab88 sp=0xc0001baa60 pc=0x450de5
zb.256lights.llc/pkg/internal/jsonrpc.(*Client).handleConn(0xc00019a040, {0xc4cfb0, 0xc00019e0f0}, {0xc4bfb8, 0xc0001a0040}, {0xc496e0, 0xc0001a0080})
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/client.go:207 +0x330 fp=0xc0001baec8 sp=0xc0001bab88 pc=0x619710
zb.256lights.llc/pkg/internal/jsonrpc.(*Client).communicate(0xc00019a040, {0xc4cfb0, 0xc00019e0f0}, 0xc0001b0010)
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/client.go:168 +0x2e7 fp=0xc0001baf90 sp=0xc0001baec8 pc=0x619347
zb.256lights.llc/pkg/internal/jsonrpc.NewClient.func1()
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/client.go:78 +0x53 fp=0xc0001bafe0 sp=0xc0001baf90 pc=0x6185d3
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001bafe8 sp=0xc0001bafe0 pc=0x47b3c1
created by zb.256lights.llc/pkg/internal/jsonrpc.NewClient in goroutine 34
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/client.go:76 +0x159

goroutine 37 gp=0xc0001888c0 m=nil [select]:
runtime.gopark(0xc0001a77b0?, 0x2?, 0x0?, 0x0?, 0xc0001a77a4?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a7648 sp=0xc0001a7628 pc=0x4734ce
runtime.selectgo(0xc0001a77b0, 0xc0001a77a0, 0xc0001a77d0?, 0x0, 0xc00019a040?, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0x7a5 fp=0xc0001a7770 sp=0xc0001a7648 pc=0x450de5
zombiezen.com/go/xcontext.CloseWhenDone.func1()
/home/light/pkg/mod/zombiezen.com/go/[email protected]/closer.go:46 +0x65 fp=0xc0001a77e0 sp=0xc0001a7770 pc=0x615de5
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a77e8 sp=0xc0001a77e0 pc=0x47b3c1
created by zombiezen.com/go/xcontext.CloseWhenDone in goroutine 36
/home/light/pkg/mod/zombiezen.com/go/[email protected]/closer.go:45 +0x105

goroutine 38 gp=0xc000188a80 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a7e38 sp=0xc0001a7e18 pc=0x4734ce
runtime.chanrecv(0xc0001820e0, 0xc0001a7ee8, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:639 +0x41c fp=0xc0001a7eb0 sp=0xc0001a7e38 pc=0x40d4bc
runtime.chanrecv2(0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/chan.go:494 +0x12 fp=0xc0001a7ed8 sp=0xc0001a7eb0 pc=0x40d092
zb.256lights.llc/pkg/zbstore.(*Codec).ReadResponse(0xc0001a0040)
/home/light/src/zb.256lights.llc/pkg/zbstore/codec.go:80 +0x33 fp=0xc0001a7f10 sp=0xc0001a7ed8 pc=0x67de73
zb.256lights.llc/pkg/internal/jsonrpc.(*Client).handleConn.func1()
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/client.go:179 +0xd3 fp=0xc0001a7fe0 sp=0xc0001a7f10 pc=0x61a153
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a7fe8 sp=0xc0001a7fe0 pc=0x47b3c1
created by zb.256lights.llc/pkg/internal/jsonrpc.(*Client).handleConn in goroutine 36
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/client.go:176 +0x105

goroutine 64 gp=0xc0000e4540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a2738 sp=0xc0001a2718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a27c8 sp=0xc0001a2738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a27e0 sp=0xc0001a27c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a27e8 sp=0xc0001a27e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 65 gp=0xc0000e4700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a2f38 sp=0xc0001a2f18 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a2fc8 sp=0xc0001a2f38 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a2fe0 sp=0xc0001a2fc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a2fe8 sp=0xc0001a2fe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 66 gp=0xc0000e48c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a3738 sp=0xc0001a3718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a37c8 sp=0xc0001a3738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a37e0 sp=0xc0001a37c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a37e8 sp=0xc0001a37e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 67 gp=0xc0000e4a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a3f38 sp=0xc0001a3f18 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a3fc8 sp=0xc0001a3f38 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a3fe0 sp=0xc0001a3fc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a3fe8 sp=0xc0001a3fe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 68 gp=0xc0000e4c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a4738 sp=0xc0001a4718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a47c8 sp=0xc0001a4738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a47e0 sp=0xc0001a47c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a47e8 sp=0xc0001a47e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 69 gp=0xc0000e4e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a4f38 sp=0xc0001a4f18 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a4fc8 sp=0xc0001a4f38 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a4fe0 sp=0xc0001a4fc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a4fe8 sp=0xc0001a4fe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 70 gp=0xc0000e4fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a5738 sp=0xc0001a5718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a57c8 sp=0xc0001a5738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a57e0 sp=0xc0001a57c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a57e8 sp=0xc0001a57e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 71 gp=0xc0000e5180 m=nil [GC worker (idle)]:
runtime.gopark(0xc0772f505fe?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc0001a5f38 sp=0xc0001a5f18 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc0001a5fc8 sp=0xc0001a5f38 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0001a5fe0 sp=0xc0001a5fc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001a5fe8 sp=0xc0001a5fe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 72 gp=0xc0000e5340 m=nil [GC worker (idle)]:
runtime.gopark(0xc0772f50aa3?, 0x1?, 0x33?, 0x95?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00050c738 sp=0xc00050c718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050c7c8 sp=0xc00050c738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00050c7e0 sp=0xc00050c7c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050c7e8 sp=0xc00050c7e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 73 gp=0xc0000e5500 m=nil [GC worker (idle)]:
runtime.gopark(0xc081a68d1b7?, 0x0?, 0x0?, 0x0?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00050cf38 sp=0xc00050cf18 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050cfc8 sp=0xc00050cf38 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00050cfe0 sp=0xc00050cfc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050cfe8 sp=0xc00050cfe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 74 gp=0xc0000e56c0 m=nil [GC worker (idle)]:
runtime.gopark(0x10c9220?, 0x1?, 0xde?, 0xad?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00050d738 sp=0xc00050d718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050d7c8 sp=0xc00050d738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00050d7e0 sp=0xc00050d7c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050d7e8 sp=0xc00050d7e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 75 gp=0xc0000e5880 m=nil [GC worker (idle)]:
runtime.gopark(0x10c9220?, 0x1?, 0x78?, 0x98?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00050df38 sp=0xc00050df18 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050dfc8 sp=0xc00050df38 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00050dfe0 sp=0xc00050dfc8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 76 gp=0xc0000e5a40 m=nil [GC worker (idle)]:
runtime.gopark(0xc081a68cc69?, 0x1?, 0xcd?, 0x27?, 0x0?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc00050e738 sp=0xc00050e718 pc=0x4734ce
runtime.gcBgMarkWorker(0xc0000b3260)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc00050e7c8 sp=0xc00050e738 pc=0x420429
runtime.gcBgMarkStartWorkers.gowrap1()
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00050e7e0 sp=0xc00050e7c8 pc=0x420305
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00050e7e8 sp=0xc00050e7e0 pc=0x47b3c1
created by runtime.gcBgMarkStartWorkers in goroutine 35
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/mgc.go:1279 +0x105

goroutine 1172 gp=0xc0001896c0 m=nil [runnable]:
modernc.org/sqlite/lib._sqlite3DbNNFreeNN(0xc0000f2000?, 0x7f0030500028?, 0x7f00315fc828?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:17459 +0xb1 fp=0xc00049dd68 sp=0xc00049dd60 pc=0x6bfc51
modernc.org/sqlite/lib._sqlite3ExprDeleteNN(0xc0000f2000, 0x7f0030500028, 0x6a9060?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:83295 +0x15c fp=0xc00049dda0 sp=0xc00049dd68 pc=0x74bddc
modernc.org/sqlite/lib._sqlite3ExprDelete(...)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:83301
modernc.org/sqlite/lib._exprListDeleteNN(0xc0000f2000, 0x7f0030500028, 0x7f0019e03828)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:84252 +0x4d fp=0xc00049ddd8 sp=0xc00049dda0 pc=0x74e40d
modernc.org/sqlite/lib._sqlite3ExprListDelete(...)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:84271
modernc.org/sqlite/lib._sqlite3ExprDeleteNN(0xc0000f2000, 0x7f0030500028, 0x10c6a50?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:83276 +0xad fp=0xc00049de10 sp=0xc00049ddd8 pc=0x74bd2d
modernc.org/sqlite/lib._sqlite3ExprDeleteNN(0xc0000f2000, 0x7f0030500028, 0x7f00309ff428?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:83271 +0xf0 fp=0xc00049de48 sp=0xc00049de10 pc=0x74bd70
modernc.org/sqlite/lib._sqlite3ExprDelete(...)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:83301
modernc.org/sqlite/lib._exprListDeleteNN(0xc0000f2000, 0x7f0030500028, 0x7f0019e03028)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:84252 +0x4d fp=0xc00049de80 sp=0xc00049de48 pc=0x74e40d
modernc.org/sqlite/lib._sqlite3ExprListDelete(...)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:84271
modernc.org/sqlite/lib._sqlite3ExprDeleteNN(0xc0000f2000, 0x7f0030500028, 0x7f0019e01828?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:83276 +0xad fp=0xc00049deb8 sp=0xc00049de80 pc=0x74bd2d
modernc.org/sqlite/lib._sqlite3ExprDelete(...)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:83301
modernc.org/sqlite/lib._exprListDeleteNN(0xc0000f2000, 0x7f0030500028, 0x7f0019e02828)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:84252 +0x4d fp=0xc00049def0 sp=0xc00049deb8 pc=0x74e40d
modernc.org/sqlite/lib._sqlite3ExprListDelete(...)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:84271
modernc.org/sqlite/lib._clearSelect(0xc0000f2000, 0x7f0030500028, 0xc0000f2000?, 0x30500028?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:118026 +0x65 fp=0xc00049df38 sp=0xc00049def0 pc=0x7abde5
modernc.org/sqlite/lib._sqlite3SelectDelete(...)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:118127
modernc.org/sqlite/lib._sqlite3SrcListDelete(0xc0000f2000, 0x7f0030500028, 0x7f00313ffc28)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:100413 +0x1b1 fp=0xc00049df80 sp=0xc00049df38 pc=0x77d5f1
modernc.org/sqlite/lib._clearSelect(0xc0000f2000, 0x7f0030500028, 0xc00049df48?, 0x49df90?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:118027 +0x7d fp=0xc00049dfc8 sp=0xc00049df80 pc=0x7abdfd
modernc.org/sqlite/lib._sqlite3SelectDelete(0xc0000f2000?, 0x7f0030505420?, 0x7f00309fee28?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:118127 +0x1d fp=0xc00049dff8 sp=0xc00049dfc8 pc=0x7ac39d
modernc.org/sqlite/lib._yy_reduce(0xc0000f2000, 0x7f003030a028, 0x54, 0x0?, {0x0?, 0x0?}, 0x7f0030505420)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:156827 +0x108a fp=0xc00049e180 sp=0xc00049dff8 pc=0x7fef6a
modernc.org/sqlite/lib._sqlite3Parser(0xc0000f2000, 0x7f003030a028, 0x1, {0x495d05?, 0x49e240?})
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:158220 +0xb1 fp=0xc00049e200 sp=0xc00049e180 pc=0x804531
modernc.org/sqlite/lib._sqlite3RunParser(0xc0000f2000, 0x7f0030505420, 0x7f0019e00020)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:160889 +0x352 fp=0xc00049e2e8 sp=0xc00049e200 pc=0x805d92
modernc.org/sqlite/lib._sqlite3Prepare(0xc0000f2000, 0x7f0030500028, 0x7f0019e00020, 0xffffffff, 0x80, 0x0, 0x7f0031729500, 0x7f00317294f0)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:117664 +0x393 fp=0xc00049e380 sp=0xc00049e2e8 pc=0x7aad13
modernc.org/sqlite/lib._sqlite3LockAndPrepare(0xc0000f2000, 0x7f0030500028, 0x7f0019e00020, 0xffffffff, 0x80, 0x0, 0x7f0031729500, 0x7f00317294f0)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:117723 +0x139 fp=0xc00049e3e8 sp=0xc00049e380 pc=0x7ab2b9
modernc.org/sqlite/lib.Xsqlite3_prepare_v3(0xc0003b3c00?, 0x691?, 0xc081a633955?, 0x2937?, 0x0?, 0xc081a63843a?, 0x10000000000?)
/home/light/pkg/mod/modernc.org/[email protected]/lib/sqlite_linux_amd64.go:117829 +0x25 fp=0xc00049e430 sp=0xc00049e3e8 pc=0x7ab7c5
zombiezen.com/go/sqlite.(*Conn).prepare(0xc0000a61e0, {0xc0003b3c00, 0x691}, 0x0)
/home/light/pkg/mod/zombiezen.com/go/[email protected]/sqlite.go:527 +0x232 fp=0xc00049e640 sp=0xc00049e430 pc=0x89fdd2
zombiezen.com/go/sqlite.(*Conn).PrepareTransient(0xc0003b3c00?, {0xc0003b3c00?, 0xb48802?})
/home/light/pkg/mod/zombiezen.com/go/[email protected]/sqlite.go:500 +0x25 fp=0xc00049e698 sp=0xc00049e640 pc=0x89fae5
zombiezen.com/go/sqlite/sqlitex.ExecuteTransientFS(0xc0000a61e0, {0xc49580?, 0xc000548aa0?}, {0xb48802, 0xb}, 0xc00049e8f0)
/home/light/pkg/mod/zombiezen.com/go/[email protected]/sqlitex/exec.go:229 +0xe5 fp=0xc00049e748 sp=0xc00049e698 pc=0x8aa805
zb.256lights.llc/pkg/internal/backend.closurePaths(0xc0000a61e0, {{0xc00045ed80?, 0xe6fcc984204f82df?}, {{0xc00011e2a0?}, {0xc00011f4f0?}}}, 0xc00011f530)
/home/light/src/zb.256lights.llc/pkg/internal/backend/backend_store.go:275 +0x69d fp=0xc00049eac8 sp=0xc00049e748 pc=0x9fd35d
zb.256lights.llc/pkg/internal/backend.(*builder).recordRealizations(0xc00039c3f0, {0xc4cf78, 0xc0001b2030}, {{}, 0x3, {0xd, 0x8e, 0x90, 0x24, 0xe3, ...}}, ...)
/home/light/src/zb.256lights.llc/pkg/internal/backend/realize.go:1319 +0x3ec fp=0xc00049ec48 sp=0xc00049eac8 pc=0xa15b2c
zb.256lights.llc/pkg/internal/backend.(*builder).do(0xc00039c3f0, {0xc4cf78, 0xc0001b2030}, {0xc000386200, 0x38}, 0xc0003c5c20)
/home/light/src/zb.256lights.llc/pkg/internal/backend/realize.go:706 +0x1987 fp=0xc00049f530 sp=0xc00049ec48 pc=0xa0b887
zb.256lights.llc/pkg/internal/backend.(*builder).realize(0xc00039c3f0, {0xc4cf78, 0xc0001b2030}, 0xc00039c360)
/home/light/src/zb.256lights.llc/pkg/internal/backend/realize.go:309 +0xd65 fp=0xc00049f9e0 sp=0xc00049f530 pc=0xa06105
zb.256lights.llc/pkg/internal/backend.(*Server).realize(0xc0000ee070, {0xc4cf78, 0xc0001b2030}, 0xc0001a1280)
/home/light/src/zb.256lights.llc/pkg/internal/backend/realize.go:80 +0x87a fp=0xc00049fc18 sp=0xc00049f9e0 pc=0xa0343a
zb.256lights.llc/pkg/internal/backend.(*Server).realize-fm({0xc4cf78?, 0xc0001b2030?}, 0xc000080cf0?)
:1 +0x36 fp=0xc00049fc48 sp=0xc00049fc18 pc=0xa28296
zb.256lights.llc/pkg/internal/jsonrpc.HandlerFunc.JSONRPC(0xad9380?, {0xc4cf78?, 0xc0001b2030?}, 0xa?)
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/server.go:40 +0x29 fp=0xc00049fc70 sp=0xc00049fc48 pc=0x61d269
zb.256lights.llc/pkg/internal/jsonrpc.ServeMux.JSONRPC(0xad9380?, {0xc4cf78, 0xc0001b2030}, 0xc0001a1280)
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/server.go:154 +0x10e fp=0xc00049fcc8 sp=0xc00049fc70 pc=0x61e00e
zb.256lights.llc/pkg/internal/backend.(*Server).JSONRPC(0xc0000ee070, {0xc4cf78, 0xc0001b2030}, 0xc0001a1280)
/home/light/src/zb.256lights.llc/pkg/internal/backend/backend.go:121 +0x273 fp=0xc00049fe40 sp=0xc00049fcc8 pc=0x9f4733
zb.256lights.llc/pkg/internal/jsonrpc.(*server).single(0xc0001aa018, {0xc4cf78, 0xc0001b2030}, {0xc48720, 0xc0000ee070}, 0xc00050bfd0?)
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/server.go:95 +0xae fp=0xc00049ff80 sp=0xc00049fe40 pc=0x61d8ce
zb.256lights.llc/pkg/internal/jsonrpc.Serve.func1()
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/server.go:82 +0x5e fp=0xc00049ffe0 sp=0xc00049ff80 pc=0x61d71e
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00049ffe8 sp=0xc00049ffe0 pc=0x47b3c1
created by zb.256lights.llc/pkg/internal/jsonrpc.Serve in goroutine 34
/home/light/src/zb.256lights.llc/pkg/internal/jsonrpc/server.go:80 +0x279

goroutine 1475 gp=0xc0002a1500 m=nil [select]:
runtime.gopark(0xc0004467b0?, 0x2?, 0xe0?, 0xc6?, 0xc00044679c?)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/proc.go:424 +0xce fp=0xc000446640 sp=0xc000446620 pc=0x4734ce
runtime.selectgo(0xc0004467b0, 0xc000446798, 0xc49360?, 0x0, 0xc48fa0?, 0x1)
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/select.go:335 +0x7a5 fp=0xc000446768 sp=0xc000446640 pc=0x450de5
zombiezen.com/go/sqlite.(*Conn).SetInterrupt.func1()
/home/light/pkg/mod/zombiezen.com/go/[email protected]/sqlite.go:303 +0x5c fp=0xc0004467e0 sp=0xc000446768 pc=0x89f1dc
runtime.goexit({})
/nix/store/mi0ybwsm6pmxzv9hsm6bcbqaq1pkf8wh-go-1.23.1/share/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0004467e8 sp=0xc0004467e0 pc=0x47b3c1
created by zombiezen.com/go/sqlite.(*Conn).SetInterrupt in goroutine 1172
/home/light/pkg/mod/zombiezen.com/go/[email protected]/sqlite.go:302 +0x206

@zombiezen zombiezen added bug Something isn't working backend Build server labels Sep 25, 2024
@zombiezen
Copy link
Collaborator Author

After auditing a bunch of code in zombiezen.com/go/sqlite and coming up empty-handed, I'm suspecting this might be golang/go#69210 based on my heavy usage of unique.Handle. Going to hold off until Go 1.23.2 to investigate further.

@zombiezen
Copy link
Collaborator Author

Upgraded to Go 1.23.2 and could no longer reproduce after 5-6 builds of simple_patch, which was usually sufficient to reproduce the problem. Closing and will reopen if I see it again.

@zombiezen zombiezen added this to the MVP milestone Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Build server bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant