Bug Report: vtgate can panic due to a race condition in the join engine primitive

[dbussink]

Overview of the Issue

When running a query with zero results on an aggregation through the join engine, it can crash due to race conditions.

This is visible today as "flakiness" in CI for the TestEmptyTableAggr test, but it's a real bug here. A previous attempt to fix this in #14012 satisfied the Go race checker, but it did not actually fix the race. There's still a classic check-then-act style race bug.

Reproduction Steps

Run the TestEmptyTableAggr tests with a higher count.

Binary Version

main

Operating System and Environment details

-

Log Fragments

panic: runtime error: index out of range [0] with length 0

goroutine 420 [running]:
vitess.io/vitess/go/vt/vtgate/engine.newAggregation({0x0?, 0x30000007d?, 0x14000b15048?}, {0x1400007ef68, 0x1, 0x1042a98c0?})
	vitess.io/vitess/go/vt/vtgate/engine/aggregations.go:352 +0x824
vitess.io/vitess/go/vt/vtgate/engine.(*ScalarAggregate).TryStreamExecute.func2(0x14000df42a0)
	vitess.io/vitess/go/vt/vtgate/engine/scalar_aggregation.go:126 +0x120
vitess.io/vitess/go/vt/vtgate/engine.(*Projection).TryStreamExecute.func1(0x14000df42a0)
	vitess.io/vitess/go/vt/vtgate/engine/projection.go:123 +0x2f4
vitess.io/vitess/go/vt/vtgate/engine.(*Join).TryStreamExecute.func1(0x14000df4230)
	vitess.io/vitess/go/vt/vtgate/engine/join.go:168 +0x638
vitess.io/vitess/go/vt/vtgate/engine.(*Route).streamExecuteShards.func1(0x0?)
	vitess.io/vitess/go/vt/vtgate/engine/route.go:345 +0x158
vitess.io/vitess/go/vt/vttablet/queryservice.(*wrappedService).StreamExecute.func1.1(0x14000dee3f0?)
	vitess.io/vitess/go/vt/vttablet/queryservice/wrapped.go:198 +0x30
vitess.io/vitess/go/vt/vttablet/grpctabletconn.(*gRPCQueryClient).StreamExecute(0x1400093e778?, {0x10429b400?, 0x14000f0f4d0?}, 0x102646c84?, {0x14000432800, 0x5c}, 0x5c6eef79?, 0x14000800000?, 0x1400093e7d8?, 0x10265aae0?, ...)
	vitess.io/vitess/go/vt/vttablet/grpctabletconn/conn.go:190 +0x168
vitess.io/vitess/go/vt/vttablet/queryservice.(*wrappedService).StreamExecute.func1({0x10429b400, 0x14000f0f4d0}, 0x14000e02a80?, {0x1042b6af8, 0x140003f3400})
	vitess.io/vitess/go/vt/vttablet/queryservice/wrapped.go:196 +0xfc
vitess.io/vitess/go/vt/vtgate.(*TabletGateway).withRetry(0x140002524d0, {0x10429b400, 0x14000f0f4d0}, 0x14000bbfb60, {0x1400093ea88?, 0x102fe94c8?}, {0x40?, 0x104198cc0?}, 0x0, 0x14000e0fd00)
	vitess.io/vitess/go/vt/vtgate/tabletgateway.go:339 +0x3d8
vitess.io/vitess/go/vt/vttablet/queryservice.(*wrappedService).StreamExecute(0x140005108b8, {0x10429b400, 0x14000f0f4d0}, 0x3b?, {0x14000432800, 0x5c}, 0x14000f0f500, 0x0, 0x0, 0x14000d070e0, ...)
	vitess.io/vitess/go/vt/vttablet/queryservice/wrapped.go:194 +0x12c
vitess.io/vitess/go/vt/vtgate.(*ScatterConn).StreamExecuteMulti.func1(0x14000f10c48, 0x0, 0x14000e28300)
	vitess.io/vitess/go/vt/vtgate/scatter_conn.go:408 +0x1e0
vitess.io/vitess/go/vt/vtgate.(*ScatterConn).multiGoTransaction.func1(0x14000f10c48, 0x1028e6f54?)
	vitess.io/vitess/go/vt/vtgate/scatter_conn.go:640 +0x138
vitess.io/vitess/go/vt/vtgate.(*ScatterConn).multiGoTransaction.func2(0x0?, 0x0?)
	vitess.io/vitess/go/vt/vtgate/scatter_conn.go:668 +0x54
created by vitess.io/vitess/go/vt/vtgate.(*ScatterConn).multiGoTransaction in goroutine 361
	vitess.io/vitess/go/vt/vtgate/scatter_conn.go:666 +0x1c8