Starboard operator crashes with panic: runtime error: index out of range [1] with length 1

[Harisanthosh]

What steps did you take and what happened:
Hi, I ran across this wonderful project recently and wanted to onboard it for one of my projects. I have tried installing Starboard operator with both Helm and Kubectl with custom CRDs. The scanning works for few seconds and then my kubernetes cluster controller crashes with panic runtime error and end up in CrashLoopBackOff state.

I also tried reducing the scope of the targeted namespaces and made the operator to run in SingleNamespace mode but of no avail. Also attaching the Debug log for reference purposes.

Debug_Log_starboard:
starboard-operator-54644fb8bf-9qkqm.log

HELM Setup:

helm install starboard-operator aqua/starboard-operator -n starboard-operator \
  --set="targetNamespaces=apps" \
  --set="trivy.ignoreUnfixed=true" \
  --set="envSecret.stringData.OPERATOR_LOG_DEV_MODE=true" \
  --set="envSecret.stringData.OPERATOR_CONFIG_AUDIT_SCANNER_ENABLED=false" \
  --set="envSecret.stringData.OPERATOR_CIS_KUBERNETES_BENCHMARK_ENABLED=false"

Kubectl Setup steps and files:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/starboard/v0.12.0/deploy/crd/vulnerabilityreports.crd.yaml \ -f https://raw.githubusercontent.com/aquasecurity/starboard/v0.12.0/deploy/crd/configauditreports.crd.yaml \ -f https://raw.githubusercontent.com/aquasecurity/starboard/v0.12.0/deploy/crd/clusterconfigauditreports.crd.yaml \ -f https://raw.githubusercontent.com/aquasecurity/starboard/v0.12.0/deploy/crd/ciskubebenchreports.crd.yaml
star-deployment.yaml.txt
star-configmap.yml.txt
star-clusterrole.yml.txt
star-rolebinding.yml.txt
star-serviceaccount.yml.txt

What did you expect to happen:
Expect the starboard operator to run without crashing. I have also updated the operators with multiple replicas to distribute the workloads however all the replicas are crashing with the same panic error.

I understand that it is not a direct issue pertaining to Starboard as it pertains to kubernetes controller crashing, however I wanted to bring it to notice to gain valuable inputs from the Aquasec community.

Thanks in advance

Environment:

• Starboard version (use starboard version): 0.12.0
• Kubernetes version (use kubectl version): 1.19.13 running in Azure Kubernetes Service
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Windows 10

[danielpacak]

👋 @Harisanthosh Thanks for the feedback and providing all the details including logs.

First off, we should handle such cases gracefully without panicking.

However, from logs it seems that the root cause of the panic is an unexpected / unrecognized type of Secret used to pull container image from a private registry. I guess you set imagePullSecrets property on a K8s workload or a service account used to run the workload as explained here.

The line of code where we panic is when we are trying to decode the base64-encoded username and password.

func (v *BasicAuth) Decode() (string, string, error) {
	bytes, err := base64.StdEncoding.DecodeString(string(*v))
	if err != nil {
		return "", "", err
	}
	split := strings.Split(string(bytes), ":")
	return split[0], split[1], nil
}

Could you provide more details about registry credentials and how you configure registry authentication? Are you using basic authentication, access token, or any other proprietary Azure solution?

[danielpacak]

Notice that running the Starboard Operator (or any K8s operator) with multiple replicas requires additional consideration. In particular enabling leader election so there's only one active replica in your cluster.

We should document this one better, but to get started see the description of the OPERATOR_LEADER_ELECTION_ENABLED variable in the Operator Configuration documentation.

[Harisanthosh]

Thanks a lot for taking time and responding to this thread @danielpacak. Yes, you are right we are currently pulling the images from a private Azure Container Registry (ACR) by giving imagePullSecrets for every workload created.

While connecting to ACR we use basic authentication at the moment where we give the container registies username and password along with the registryURI and use it as a service connection.

Thank you for also pointing me out regarding the leader election, let me try the same as well.