Conformance GitHub action sometimes panics

[SuperSandro2000]

We are running the conformance github action of part of our registry CI. Sometimes it fails randomly and succeeds with the next unrelated commit again.
We are running the conformance tests in our own CI, too, where this is not happening. I am not familiar with ginkgo and currently I don't have a good idea how to debug this.

•! Panic [0.000 seconds]
OCI Distribution Conformance Tests
/go/src/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Pull
  /go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:13
    Setup
    /go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:17
      Populate registry with test blob [It]
      /go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:18

      Test Panicked
      runtime error: invalid memory address or nil pointer dereference
      /usr/local/go/src/runtime/panic.go:199

      Full Stack Trace
      github.com/bloodorangeio/reggie.(*Response).GetRelativeLocation(0x0, 0xc00001e060, 0x0)
      	/go/pkg/mod/github.com/bloodorangeio/[email protected]/response.go:22 +0x3a
      github.com/opencontainers/distribution-spec/conformance.glob..func1.1.1.1()
      	/go/src/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:[23](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:24) +0xbc
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc00008e9c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:113 +0xb8
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc00008e9c0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/runner.go:64 +0xcf
      github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc00009dce0, 0x9eb060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/leafnodes/it_node.go:[26](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:27) +0x64
      github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc000346000, 0x0, 0x9eb060, 0xc000070b80)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:215 +0x5b5
      github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc000346000, 0x9eb060, 0xc000070b80)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/spec/spec.go:138 +0x101
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0000b6840, 0xc000346000, 0x0)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:200 +0x10f
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0000b6840, 0x1)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:170 +0x120
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0000b6840, 0xc0002fca78)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/specrunner/spec_runner.go:66 +0x117
      github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0000f59d0, 0x7f9a[27](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:28)6e72a8, 0xc0000c6600, 0x943418, 0x22, 0xc000170690, 0x3, 0x3, 0x9f58a0, 0xc000070b80, ...)
      	/go/pkg/mod/github.com/onsi/[email protected]/internal/suite/suite.go:79 +0x586
      github.com/onsi/ginkgo.RunSpecsWithCustomReporters(0x9eb720, 0xc0000c6600, 0x943418, 0x22, 0xc000071900, 0x3, 0x4, 0x4)
      	/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:2[29](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:30) +0x217
      github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters(0x9eb720, 0xc0000c6600, 0x94[34](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:35)18, 0x22, 0xc0000fff40, 0x2, 0x2, 0x3651cab1)
      	/go/pkg/mod/github.com/onsi/[email protected]/ginkgo_dsl.go:217 +0xad
      github.com/opencontainers/distribution-spec/conformance.TestConformance(0xc0000c6600)
      	/go/src/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:21 +0x18b
      testing.tRunner(0xc0000c6600, 0x9627c8)
      	/usr/local/go/src/testing/testing.go:909 +0xc9
      created by testing.(*T).Run
      	/usr/local/go/src/testing/testing.go:960 +0x[35](https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220#step:7:36)0

https://github.com/sapcc/keppel/actions/runs/3821378134/jobs/6500467220

[sudo-bmitch]

Looks like the POST to blob/uploads failed, and we don't have error handling on the call in the currently released branch. There is error handling on the repo main branch. (See #356.)

The intermittent failure points to a race condition, likely starting the conformance test before the registry is fully started. You may want to use some kind of wait-for-it like tool to pause until the server is running.

[SuperSandro2000]

I added a wait condition on our healthcheck endpoint and that improved things.