panic "close of closed channel" during temporary directory cleanup

[chkohner]

What steps did you take and what happened:

Periodically Octant crashes with a panic. The amount of time is indeterminate. Sometimes it runs fine for ~45min or more, sometimes it fails to start entirely. It usually will start up again immediately if run after crashing.

What did you expect to happen:

Octant should run until explicitly terminated.

Anything else you would like to add:

The stack trace printed to the console looks something like this:

2022-04-13T09:02:18.099-0700    INFO    cluster/cluster.go:126  removing cluster client temporary directory     {"component": "cluster client"}
2022-04-13T09:02:18.892-0700    INFO    config/dash.go:182      updated kube config context     {"new-kube-context": ""}2022-04-13T09:02:18.892-0700    INFO    cluster/cluster.go:126  removing cluster client temporary directory     {"component": "cluster client"}
E0413 09:02:18.900363   28164 runtime.go:78] Observed a panic: "close of closed channel" (close of closed channel)
goroutine 43 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x232b120, 0x956dc60})
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xfffffffe})
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x232b120, 0x956dc60})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000061680)
        /__w/octant/octant/vendor/k8s.io/client-go/tools/cache/shared_informer.go:752 +0x287
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88
panic: close of closed channel [recovered]
        panic: close of closed channel

goroutine 43 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xfffffffe})
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x232b120, 0x956dc60})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc000061680)
        /__w/octant/octant/vendor/k8s.io/client-go/tools/cache/shared_informer.go:752 +0x287
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /__w/octant/octant/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88

Environment:

• Octant version (use octant version): 0.25.1
• Kubernetes version (use kubectl version): Client: 1.22.5 (windows) / Server: 1.21.9 (linux)
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Windows 11

[chkohner]

In case it's useful to anyone, I wound up using this script to let me keep using my existing browser session through crashes. 😢

#!/bin/sh

DISABLE_BROWSER=""
while true
do
    octant ${DISABLE_BROWSER}
    EXITCODE=$?
    if [ ${EXITCODE} != 2 ]; then # loop only if go's panic exit code
        exit ${EXITCODE}
    fi
    # Disable browser for subsequent loops
    DISABLE_BROWSER=--disable-open-browser
    echo "Octant crashed! Restarting..."
done