Project Quota map is not thread-safe (concurrent map read and map write error)

[Ezetowers]

Summary

While using cri-o as container runtime, a race condition was found by the Golang compiler and the service crashed. It seems to be the error is associated with the lack of locking when the quota map is accessed in the projectquota.go file in this library.

Oct 17 16:50:01 den-r25-u03 crio[5299]: fatal error: concurrent map read and map write
Oct 17 16:50:01 den-r25-u03 crio[5299]: goroutine 4359466 [running]:
Oct 17 16:50:01 den-r25-u03 crio[5299]: runtime.throw({0x557161bfc6b2, 0xc0006622e0})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /usr/lib64/go-1.17/src/runtime/panic.go:1198 +0x71 fp=0xc0042ab218 sp=0xc0042ab1e8 pc=0x5571604e8a31
Oct 17 16:50:01 den-r25-u03 crio[5299]: runtime.mapaccess2_faststr(0x7f5e602f0fa0, 0x0, {0xc000c1e9a0, 0x6b})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /usr/lib64/go-1.17/src/runtime/map_faststr.go:116 +0x3d4 fp=0xc0042ab280 sp=0xc0042ab218 pc=0x5571604c5d94
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/containers/storage/drivers/quota.(*Control).fsDiskQuotaFromPath(0xc0039aadc0, {0xc000c1e9a0, 0x6b})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/github.com/containers/storage/drivers/quota/projectquota.go:270 +0xb2 fp=0xc0042ab3d8 sp=0xc0042ab280 pc=0x557160cbcad2
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/containers/storage/drivers/quota.(*Control).GetDiskUsage(0xc002bec6c0, {0xc000c1e9a0, 0x40}, 0xc0022d0740)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/github.com/containers/storage/drivers/quota/projectquota.go:257 +0x3e fp=0xc0042ab550 sp=0xc0042ab3d8 pc=0x557160cbc95e
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/containers/storage/drivers/overlay.(*Driver).ReadWriteDiskUsage(0xc002bec6c0, {0xc002953cab, 0x40})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/github.com/containers/storage/drivers/overlay/overlay_cgo.go:17 +0x72 fp=0xc0042ab5a8 sp=0xc0042ab550 pc=0x557160cda132
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/internal/lib/stats.(*StatsServer).writableLayerForContainer(0xc000626e40, 0xc000513180)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/internal/lib/stats/stats_server.go:193 +0x1af fp=0xc0042ab668 sp=0xc0042ab5a8 pc=0x557161b1402f
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/internal/lib/stats.(*StatsServer).populateWritableLayer(0x100557160c1e426, 0xc0037349f0, 0xc00005a028)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/internal/lib/stats/stats_server.go:174 +0x25 fp=0xc0042ab6b8 sp=0xc0042ab668 pc=0x557161b13de5
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/internal/lib/stats.(*StatsServer).updateContainer(0xc000626e40, 0xc000513180, 0xc0017b7340)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/internal/lib/stats/stats_server.go:147 +0x195 fp=0xc0042ab748 sp=0xc0042ab6b8 pc=0x557161b13bf5
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/internal/lib/stats.(*StatsServer).statsForContainer(0xc000626e78, 0xc000a9c380, 0x10000c000e08640)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/internal/lib/stats/stats_server.go:322 +0x7b fp=0xc0042ab780 sp=0xc0042ab748 pc=0x557161b154db
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/internal/lib/stats.(*StatsServer).StatsForContainers(0xc000626e40, {0xc00338c108, 0x1, 0xc0042ab8d0})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/internal/lib/stats/stats_server.go:311 +0x185 fp=0xc0042ab878 sp=0xc0042ab780 pc=0x557161b15205
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/server.(*Server).ListContainerStats(0xc00040a000, {0x557162580980, 0xc0048ca150}, 0xc0034638a0)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/server/container_stats_list.go:30 +0x149 fp=0xc0042ab918 sp=0xc0042ab878 pc=0x557161b7d8a9
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/server/cri/v1alpha2.(*service).ListContainerStats(0xc0048ca150, {0x557162580980, 0xc0048ca150}, 0xc0042ab9f0)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/server/cri/v1alpha2/rpc_list_container_stats.go:14 +0x26 fp=0xc0042ab948 sp=0xc0042ab918 pc=0x557161ba4ca6
Oct 17 16:50:01 den-r25-u03 crio[5299]: k8s.io/cri-api/pkg/apis/runtime/v1alpha2._RuntimeService_ListContainerStats_Handler.func1({0x557162580980, 0xc0048ca150}, {0x55716242dfe0, 0xc0034638a0})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:9226 +0x78 fp=0xc0042ab988 sp=0xc0042ab948 pc=0x557161a32098
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/internal/log.UnaryInterceptor.func1({0x557162580980, 0xc0048ca0c0}, {0x55716242dfe0, 0xc0034638a0}, 0xc00352ba40, 0xc0038cc030)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/internal/log/interceptors.go:56 +0xc3 fp=0xc0042aba10 sp=0xc0042ab988 pc=0x5571612b3523
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x557162580980, 0xc0048ca0c0}, {0x55716242dfe0, 0xc0034638a0})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc0042aba50 sp=0xc0042aba10 pc=0x557161ba599a
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/cri-o/cri-o/server/metrics.UnaryInterceptor.func1({0x557162580980, 0xc0048ca0c0}, {0x55716242dfe0, 0xc0034638a0}, 0xc001ac6000, 0xc001ac60e0)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/server/metrics/interceptors.go:24 +0xb2 fp=0xc0042abad8 sp=0xc0042aba50 pc=0x55716196d5f2
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x557162580980, 0xc0048ca0c0}, {0x55716242dfe0, 0xc0034638a0})
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25 +0x3a fp=0xc0042abb18 sp=0xc0042abad8 pc=0x557161ba599a
Oct 17 16:50:01 den-r25-u03 crio[5299]: github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x557162580980, 0xc0048ca0c0}, {0x55716242dfe0, 0xc0034638a0}, 0xc00352bbb8, 0x55716227e260)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34 +0xbf fp=0xc0042abb70 sp=0xc0042abb18 pc=0x557161ba583f
Oct 17 16:50:01 den-r25-u03 crio[5299]: k8s.io/cri-api/pkg/apis/runtime/v1alpha2._RuntimeService_ListContainerStats_Handler({0x5571624b8640, 0xc0005a8278}, {0x557162580980, 0xc0048ca0c0}, 0xc001384060, 0xc000738b70)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go:9228 +0x138 fp=0xc0042abbc8 sp=0xc0042abb70 pc=0x557161a31f58
Oct 17 16:50:01 den-r25-u03 crio[5299]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001fc8c0, {0x5571625ba668, 0xc000346c00}, 0xc001013440, 0xc0004e35f0, 0x5571635a7d48, 0x0)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/google.golang.org/grpc/server.go:1282 +0xccf fp=0xc0042abe48 sp=0xc0042abbc8 pc=0x557160edd78f
Oct 17 16:50:01 den-r25-u03 crio[5299]: google.golang.org/grpc.(*Server).handleStream(0xc0001fc8c0, {0x5571625ba668, 0xc000346c00}, 0xc001013440, 0x0)
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/google.golang.org/grpc/server.go:1616 +0xa2a fp=0xc0042abf68 sp=0xc0042abe48 pc=0x557160ee156a
Oct 17 16:50:01 den-r25-u03 crio[5299]: google.golang.org/grpc.(*Server).serveStreams.func1.2()
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/google.golang.org/grpc/server.go:921 +0x98 fp=0xc0042abfe0 sp=0xc0042abf68 pc=0x557160edb2b8
Oct 17 16:50:01 den-r25-u03 crio[5299]: runtime.goexit()
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /usr/lib64/go-1.17/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0042abfe8 sp=0xc0042abfe0 pc=0x55716051c921
Oct 17 16:50:01 den-r25-u03 crio[5299]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 17 16:50:01 den-r25-u03 crio[5299]:         /home/abuild/rpmbuild/BUILD/cri-o-1.23.3/vendor/google.golang.org/grpc/server.go:919 +0x294

Since this is a race condition, I am not able to create a set of steps to reproduce this issue, but if you think this problem belongs to this lib I can create a PR adding proper locking to the quota maps in the projectquota.go file

Kind Regards
Ezequiel

[haircommander]

Thanks! I was trying to see if there was any other thing we should think about. @Ezetowers if you'd like to add a lock to the Control object, and lock the couple of public functions, I would appreciate it